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EXECUTIVE  SUMMARY 


One  of  the  first  evaluation  steps  of  a  training  system  should  be  to  establish  user  acceptance  and 
determine  how  the  operators  visualize  potential  training  utility.  One  of  the  more  promising 
recent  military  training  systems  is  Distributed  Mission  Operations  (DMO)  training.  DMO 
training  systems  today  typically  consist  of  networked  simulators  to  allow  for  multi-player 
training  on  realistic  combat  exercises.  The  work  discussed  in  this  report  focuses  on  the  user 
acceptance  of  the  Air  Force  Research  Laboratory,  Human  Effectiveness  Directorate,  Warfighter 
Training  Research  Division’s  (AFRL/HEA’s)  DMO  training  research  site  in  Mesa,  AZ. 
Specifically,  how  do  the  users  perceive  the  value  of  the  Mesa  DMO  system  and  what  mission 
essential  experiences  is  the  Mesa  DMO  system  perceived  as  providing  valuable  training?  This 
work  sought  to  document  the  F-16  and  Airborne  Warning  and  Control  Station  (AWACS)  user 
opinion  results  in  detail  and  provide  the  necessary  summary  results  for  the  larger  DMO  training 
effectiveness  study  documented  in  AFRL-HE-AZ-TR-2006-0015,  Volume  I,  Distributed  Mission 
Operations  Within-Simulator  Training  Effectiveness  Baseline  Study:  Summary  Report. 

Users  based  their  opinions  on  experiences  obtained  in  the  Mesa  DMO  training  research 
environment.  All  327  F-16  and  49  AWACS  participants  experienced  a  consistent  local  area 
network  five-day  training  research  syllabus  containing  over  40  total  4vX  scenarios,  primarily  air- 
to-air  with  some  air-to-ground.  In  conjunction  with  a  computer-generated  threat  system  and  an 
instructor  operator  station,  the  Mesa  DMO  research  environment  consisted  of  four  high-fidelity 
F-16  simulators,  one  high-fidelity  AWACS  simulator,  and  a  full  complement  of  high  fidelity 
brief/debrief  equipment  (including  time-stamped  replays  of  avionic  equipment  and 
communication). 

Examining  the  user  ratings  for  opinions  about  the  DMO  environment  revealed  that  across  all  58 
rated  statements,  both  F-16  and  AWACS  participants  generally  like  the  environment.  “I  would 
recommend  this  training  experience  to  other  pilots/controllers”  was  rated  by  all  but  one  of  49 
controllers  and  all  but  16  of  327  pilots  with  the  highest  rating  possible  of  “Strongly  Agree.” 
Performing  a  content  analysis  on  their  open-ended  responses  to  what  they  felt  was  most 
beneficial  about  the  environment,  F-16  pilots  most  frequently  wrote  comments  relating  to 
“realistic  qualities,”  while  AWACS  operators  most  frequently  wrote  positive  comments 
regarding  the  “scenarios.”  The  second  most  written  comment  for  both  the  F-16  and  AWACS 
operators  related  to  positive  skill  acquisition.  Finally,  when  asked  to  rate  to  what  extent  the 
Mesa  DMO  system  provides  the  45  different  F-16  critical  air-to-air  experiences  (defined  by  the 
Mission  Essential  Competency  process),  the  pilots  rated  the  Mesa  DMO  higher  than  all  seven 
other  environments  surveyed.  Pilot  rated  that  38  experiences  (84%)  could  be  gained  “to  a 
moderate  extent”  or  higher  in  the  Mesa  DMO  environment,  and  17  experiences  (38%)  were  rated 
“to  a  substantial  extent”  or  higher. 


v 


ACKNOWLEDGEMENTS 


The  opinions  expressed  within  this  report  are  those  of  the  authors  and  do  not  necessarily  reflect 
the  views  of  the  sponsoring/employing  organizations. 

This  work  was  funded  by  the  U.S.  Air  Force  under  Contract  #F41624-97-D-5000.  The  authors 
wish  to  thank  the  following  individuals  for  their  significant  contribution  (in  alphabetical  order): 

Dr.  George  Alliger, 

Michelle  Borlase, 

Bob  Chapman, 

Chuck  “DBear”  Colegrove, 

Jim  Hartman, 

Mary  Johnson, 

Stephen  “Blaster”  Masters, 

Chris  “New  York”  Metz, 

Antoinette  Portrey, 

Margaret  Schurig, 

LtCol.  Steve  “Simple”  Symons, 

Traci  Smith, 

Dr.  William  Stock, 

Michael  “Booma”  Wang, 
and 

Mark  “Pop”  Warner. 


vi 


DISTRIBUTED  MISSION  OPERATIONS  WITHIN-SIMULATOR 
TRAINING  EFFECTIVENESS  BASELINE  STUDY:  PARTICIPANT  UTILITY 
AND  EFFECTIVENESS  OPINIONS  AND  RATINGS,  VOLUME  IV 


INTRODUCTION 


When  evaluating  a  simulation  training  device,  researchers  may  rely  on  a  number  of  different  data 
sources  to  assess  training  utility  and  overall  effectiveness.  One  of  the  most  common  sources  to 
rely  on  is  participant  opinion  (i.e.,  user  survey)  data.  Training  research  practitioners  concur  that 
user  opinion  does  not  bestow  an  adequate  measure  of  training  success  or  effectiveness,  but  it  can 
establish  user  acceptance,  which  is  often  a  necessary  condition  for  continued  system  use, 
development,  and/or  further  effectiveness  research  (Bell  &  Waag,  1998;  Salas,  Bowers,  & 
Rhodenizer,  1998;  Stanton,  1996).  Therefore,  one  of  the  first  evaluation  steps  of  a  training 
system  should  be  to  establish  user  acceptance  and  determine  how  the  operators  visualize 
potential  training  utility. 

One  of  the  more  promising  recent  military  training  systems  is  Distributed  Mission  Operations 
(DMO)  training.  DMO  training  systems  today  typically  consist  of  networked  simulators  to  allow 
for  multi-player  training  on  realistic  combat  exercises.  Significantly  different  than  stand-alone 
simulation  systems  that  train  primarily  emergency  procedures  or  part-tasks,  these  networked 
simulation  environments  provide  entire  combat-like  experiences  involving  the  real-time 
interaction  with  other  simulated  entities,  both  friendly  and  foe.  This  allows  for  higher  order  skill 
development  and  teamwork  coordination  while  executing  significant  portions  of  an  entire 
mission.  Before  DMO,  training  opportunities  for  these  larger  scope  experiences  could  rarely  be 
gained  outside  of  war.  Some  DMO  environment  examples  within  the  United  States  Air  Force 
(USAF)  include  Shaw  Air  Force  Base  (AFB),  Eglin  AFB,  Mountain  Home  AFB,  and  the  Air 
Force  Research  Laboratory’s  training  research  site  in  Mesa,  AZ  (AFRL/HEA).  The  current  work 
focuses  on  the  user  acceptance  of  AFRL/HEA’ s  DMO  training  research  site  which  contains  four 
high-fidelity  F-16  simulators,  an  AWACS  simulator,  and  a  threat  generation  system.  Naturally, 
we  must  then  ask,  “How  do  the  users  perceive  the  system  as  a  training  device?”  And,  “Which 
mission  essential  experiences  do  the  users  expect  a  system  like  Mesa’s  to  provide?”  Specifically 
regarding  the  Mesa  DMO  training  research  site,  we  now  discuss  each  question  and  its  relevant 
literature  in  turn. 

How  do  the  users  perceive  the  Mesa  DMO  system? 

A  number  of  historical  user  acceptance  results  on  very  early  multiplayer  simulation 
environments  might  lead  us  to  conclude  that  users  would  be  generally  positively  disposed 
towards  such  training  technologies  (e.g.,  Thomas,  Houck,  &  Bell,  1990;  Houck,  Thomas,  &  Bell, 
1991;  Waag  &  Houck,  1994).  However,  negative  opinion  study  results  are  not  usually  published 
and  the  associated  negatively-viewed  training  systems  quietly  fade  into  obscurity,  hence  the  need 
to  routinely  perfonn  user  acceptance  research  on  new  or  changing  systems.  Since  DMO  systems 
have  been  in  existence  a  number  of  years  and  the  contemporary  versions  have  grown 
tremendously  in  popularity,  the  acceptance  does  seem  somewhat  assured;  it  almost  goes  without 
saying  that  observation  among  DMO  participants,  expert  observers,  and  the  DMO  community  in 
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general  appears  to  indicate  overall  DMO  acceptance.  We  would  expect  the  same  of  the  Mesa 
Research  Site.  Recent  DMO  user  opinion  research,  albeit  only  published  work,  validates  this 
impression  with  warfighters  and  falls  in-line  with  the  older  multiplayer  simulation  opinion 
results  (Chapel,  2000;  Crane,  Robbins,  &  Bennett,  2000;  Crane,  Schiflett,  &  Oser,  2000).  These 
three  studies  all  examined  the  Mesa  DMO  training  research  site,  and  positive  user  opinions  were 
given  for  the  air  combat  DMO  experiences,  including  specific  positive  feedback  regarding 
training  potential  for  skills  such  as  communication,  situation  awareness,  teamwork,  tactical 
execution,  leadership,  briefing,  beyond-visual-range  tactics,  decision  making,  and  radar 
mechanization.  We  must  remember,  however,  that  each  DMO  system  contains  a  unique  set  of 
interconnected  technologies — the  hardware,  software,  and  physical/functional  fidelity  levels 
drastically  differ  depending  on  DMO  location  and,  over  time,  these  systems  vary  within  each 
DMO  location.  The  Mesa  Research  Site,  unsurprisingly,  has  been  upgraded  over  time. 

Inasmuch  as  the  DMO  systems  vary,  so  too  may  user  perceptions  vary.  DMO  user  opinion 
research  should  be  conducted  for  each  site  and  repeated  when  major  physical  or  functional 
changes  occur  either  in  the  DMO  training  environment  or  in  the  environment  the  DMO  system  is 
attempting  to  replicate. 

Which  mission  essential  experiences  are  provided? 

With  numerous  simulations  networked  together,  DMO  affords  very  unique  training.  Unlike 
academics,  part-task  trainers,  or  stand-alone  simulators,  with  DMO  the  training  community  can 
combine  essential  elements  of  battle  into  a  single  environment;  that  is,  we  can  more  closely 
replicate  entire  portions  of  battle  without  using  actual  assets.  Warfighters  therefore  are  provided 
the  chance  to  bring  together  various  skills  and  knowledge  in  combat-like  missions,  potentially 
providing  them  with  the  opportunity  to  obtain  mission  essential  experiences  once  only  gained 
during  actual  war  or  infrequent  range  exercises.  Unfortunately,  until  recently,  the  research 
community  did  not  possess  a  list  of  these  mission  essential  warfighter  experiences,  and  the  prior 
user  opinion  research  did  not  anchor  to  a  common  theoretical  framework.  As  to  be  expected, 
user  opinion  research  typically  resulted  from  training  research  practitioners  developing 
idiosyncratic  surveys  to  evaluate  each  system,  with  a  general  theme  revolving  around  perceived 
likes,  dislikes,  pros,  cons,  “sim-isms,”  etc.  Though  these  surveys  do  provide  valuable 
information,  ideally  warfighters  for  given  mission  areas  could  be  brought  together  and  critical 
training  experiences  elicited  from  them  in  a  deliberate  and  systematic  manner;  these  warfighter- 
defined  critical  training  experiences  could  then  serve  as  the  constructs  on  user  training  utility 
surveys,  revealing  the  utility  of  the  DMO  system  to  provide/train  each  experience.  Fortunately, 
the  Mission  Essential  Competency  (MEC)  process  (Colegrove  &  Alliger,  2002)  is  a  systematic 
process  that  elicits  this  information  from  operational  warfighters.  Of  particular  relevance  to  the 
current  work,  the  MEC  process  requires  that  operational  warfighters  define  the  critical 
experiences  necessary  for  the  warfighter  to  solidify  knowledge  and  skills  and  therefore  be  fully 
prepared  to  perform  his/her  duties  in  a  non-pennissive  environment.  Table  1  is  a  list  of  the  F-16 
air-to-air  combat  critical  experiences. 
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Table  1  The  45  F-16  air-to-air  critical  experiences  defined  by  operational  pilots  as  part  of  the  MEC  process. 


Restrictions  to  visibility  (e.g.,  haze) 

Restricted  weapons  load  (e.g.,  due  to  previous  weapons  employment, 
incompleted  reload,  or  WRM  limitations) 

Visual  illusions 

Limited  fuel  remaining  (e.g.,  due  to  increased  fuel  consumption,  low  fuel 
remaining,  lack  of  tanker  support,  or  the  inability  to  inflight  refuel) 

Marginal/minimal  cloud  clearance 

Operating  area  restrictions  (e.g.,  geographic,  altitude,  or  political) 

Daytime  employment 

Degraded  comm,  (e.g.,  due  to  ownship  systems  malfunction,  another  aircraft's 
malfunction,  or  the  inability  to  use  HQ  or  secure  voice  by  one  or  more  aircraft) 

Dusk  employment 

Formation  responsibilities  (e.g.,  position,  visual  lookout) 

Night  employment 

Degraded  weapons  employment  capability  (e.g.,  fire  control,  RWR,  missiles) 

Dynamic  retasking/scramble 
operations 

Battle  damage  (e.g.,  operations  with  battle  damage  of  ownship  or  of  another 
aircraft  in  the  formation) 

G-induced  physical  limitations 

Supersonic  employment 

Limited  time  to  act/react  to  situation 

A  full  range  of  adversary  air  threat  and  mix  (e.g.,  various  limitations  in 
maneuvering,  tactics,  and  weapons) 

Operations  with  friendly  IADS 

A  full  range  of  adversary  ground  type  and  mix  (including  old  and  latest  threats) 

Task  saturation 

Various  employment  altitudes  (e.g.,  low,  med,  high) 

Degraded  Nav 

Ownship  and  friendly  electronic  counter  measures 

Radar  search  responsibilities 

Operations  against  a  threat  using  chaff/flare 

Lost  mutual  support 

Using  chaff/flare  to  deny/defeat  enemy  radar/weapons. 

Air  refueling 

Operations  against  adversary  comm  jam/spoofmg 

Emergency  procedures 

Operations  against  air  or  ground  adversary  jamming 

Targeting  and  sorting  responsibilities 

ROE  limitations  (e.g.,  operations  in  an  environment  that  has  restrictive  ID 
requirements  (other  than  BVR  weapons  free)  such  as  VID,  PID,  hostile 
declaration  required  by  offboard  sources) 

Mountainous  terrain 

Fatigue/time  on  task  (e.g.,  long  range  force  employment) 

1 : 1  Force  ratio  (e.g.,  2  v  2,  4  v  4) 

Various  initial  conditions  (e.g.,  perch  setups,  CAP  &  tap,  and  flights  on  ranges) 

1:2  Force  ratio  (e.g.,  4  v  8) 

Simulated  weapons  employment  (e.g.,  training) 

1 :3+  Force  ratio  (e.g.,  4  v  12+) 

Live  weapons  employment  (e.g.,  WSEP,  combat) 

OCA  escort  missions 

Employment  with  various  package  requirements  (e.g.,  operations  in  conjunction 
with  different  aircraft  and  supporting  assets,  includes  determining  and  operating 
with  minimum  package  requirements) 

OCA  sweep  missions 

A  DMO  environment,  as  judged  by  warfighters,  should  provide  many  of  these  warfighter- 
defined  experiences.  For  longer  tenn  research  initiatives,  the  MEC  critical  experiences  fonn  a 
favored  framework  for  formally  mapping  DMO  user  opinion  research. 


CURRENT  WORK 


For  overall  training  effectiveness  of  the  Mesa  DMO  training  research  site,  (Schreiber  &  Bennett, 
2006)  reported  a  DMO  training  effectiveness  study.  In  that  report,  the  authors  collected  data  on 
over  3,000  many  versus  many  air  combat  engagements.  The  authors  reported  numerous  different 
data  sources  converging  on  the  highly  positive  in-simulator  training  effectiveness  of  the  Mesa 
DMO  research  environment.  As  such,  that  report’s  focus  was  to  document  the  overall  results 
from  the  central  hypotheses  of  each  dataset  to  establish  within-simulator  training  effectiveness. 

In  direct  support  of  that  overall  study,  two  distinct  user  opinion  data  collection  endeavors  were 
analyzed  for  the  current  work.  The  first  effort  involved  aggregating  and  analyzing  a  legacy 
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DMO  user  opinion  survey  collected  from  Mesa  DMO  warfighters  during  their  week  of 
participating  in  DMO  training  research.  From  continued  use  since  January  2002,  this  survey 
produced  a  sizeable  dataset  for  analysis.  The  second  effort  involved  leveraging  a  portion  of  the 
MEC  data  collection  process.  Specifically,  to  what  extent  operational  warfighters  perceive  that 
Mesa  DMO  can  afford  providing  specific  MEC  experiences. 

General  goals  for  this  work  sought  to  (a)  report  the  survey  opinion  results  in  detail,  (b)  leverage 
the  new  MEC  process  and  document  what  MEC  experiences  the  users  feel  DMO  is  best  suited 
for,  and  (c)  provide  the  summary  results  for  the  larger  Schreiber  and  Bennett,  (2006)  DMO 
training  effectiveness  study.  More  specifically,  we  investigated  the  following  hypotheses  in 
support  of  the  abovementioned  goals: 

1 .  Pilot  participants  will  provide  favorable  ratings  of  the  environment. 

2.  AWACS  participants  will  provide  favorable  ratings  of  the  environment. 

3.  Due  to  occasional,  but  significant  enhancements  to  the  Mesa  DMO  environment,  we 
expect  a  significant  corresponding  increase  in  ratings  over  time. 

4.  We  expect  pilots  to  rate  that  the  environment  provides  many  of  the  essential  experiences 
as  defined  by  the  MEC  process. 


METHODS 


Mesa  DMO  Training  Research  Environment. 

A  portion  of  the  following  infonnation  is  from  General  Method  in  Schreiber  and  Bennett  (2006). 

Users  based  their  opinions  on  experiences  obtained  in  the  Mesa  DMO  training  research 
environment.  The  Mesa  DMO  research  environment  operated  predominantly  as  a  stand-alone 
local  area  network  (LAN),  but  about  15%  of  DMO  exercises  over  the  data  collection  period  in 
the  past  several  years  were  part  of  a  larger,  wide-area  network  (WAN)  exercise.  Of  the  users 
included  in  the  current  user  opinion  research  work,  all  of  them  experienced  a  consistent  LAN 
five-day  training  research  syllabus  containing  over  40  total  4vX  scenarios,  primarily  air-to-air 
with  some  air-to-ground.  In  conjunction  with  a  computer  generated  threat  system  and  an 
instructor  operator  station  (IOS),  the  Mesa  DMO  research  environment  (Figure  1)  consisted  of 
four  high-fidelity  F-16  simulators  and  one  high-fidelity  Airborne  Warning  and  Control  System 
(AWACS)  simulator.  The  F-16s,  AWACS,  and  threat  entities  interoperated  according  to 
Distributed  Interactive  Simulation  (DIS)  standards  (IEEE  Standard  for  Distributed  Interactive 
Simulation  -  Application  Protocols,  1995)  version  4.02  or  version  6.0.  Pictures  of  the  overall 
environment,  one  F-16  simulator,  the  AWACS  simulator,  and  the  brief/debrief  facility  are 
provided  in  Figures  1-4,  respectively. 
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Figure  1  Mesa  DMO  training  research  environment. 


The  high-fidelity  F-16  Block  30  simulators  utilized  360  degree  out-the-window  visual  displays 
with  either  SGI  Onyx  II  Reality  Monsters  or  PC  Nova  IIs  running  Aechelon  runtime  software 
(Figure  2).  The  visual  system  used  high  resolution  photo-realistic  databases  of  the  Sonoran 
desert  overlaid  on  terrain  elevation  data  of  the  region.  The  hardware  was  very  nearly  identical  to 
that  found  in  the  actual  F-16,  as  was  the  software  (Software  Capabilities  Upgrade  (SCU)  version 
4).  Depending  on  the  type  of  mission  to  be  flown,  F-16  weapon  load-outs  for  missions  usually 
consisted  of  differing  combinations  of  the  gun,  the  Air  Intercept  Missile  (AIM-9),  the  Advanced 
Medium  Range  Air-to-Air  Missile  (AMRAAM),  and/or  the  Mk-82  and  Mk-84  general  purpose 
bombs.  A  high-fidelity  Solipsys  version  6  AWACS  sensor  simulation  was  also  used  to  provide  a 
more  realistic  environment  (Figure  3). 

The  Automated  Threat  Engagement  System  (ATES)  generated  the  adversaries.  ATES  is  a 
computerized,  real-time  threat  generation  system  that  operates  on  standard  DIS  networks, 
providing  air-to-air,  air-to-ground,  and  surface-to-air  threats.  The  ATES  system  incorporates 
aerodynamic  modeling,  atmospheric  models,  radar  models,  infrared  models,  and  data  parameter 
tables  for  thrust,  drag,  lift,  etc.  For  the  current  work,  most  threat  air  models  were  the  MiG-29, 
MiG-27/23,  and  Su-27  loaded  with  the  AA-8,  AA-lOa  and  AA-lOc  air-to-air  missiles.  Ground 
threats  usually  included  the  SA-2,  SA-6,  and  SA-8,  and  AAA.  Threat  aircraft  typically 
performed  maneuvers  and/or  scripted  flight  paths  while  reacting  to  the  F-16’s  maneuvers  and 
weapons. 
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Figure  2  Interior  view  of  one  of  the  four  high-fidelity  F-16  simulators. 


Figure  3  Close-up  view  of  the  AWACS  simulator. 
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The  current  debrief  facility  includes  six  50-inch  plasma  screens  —  one  for  a  God’s  eye  view,  one 
for  AWACS,  and  one  dedicated  for  each  of  the  four  F-16s.  Each  of  the  F-16  plasma  screens 
presented  four  avionic  displays  from  the  F-16  (Figure  4).  The  time  synchronized  replay  included 
all  communications  and  could  be  paused,  fast-forwarded,  or  rewound  according  to  the  lead 
pilot’s  desired  use  of  the  allotted  debrief  time. 


Figure  4  Brief/debrief  room. 

As  a  training  research  installation  striving  to  continually  integrate  and  evaluate  new  training 
technologies,  personnel  at  the  DMO  site  at  Mesa  enhanced  the  simulation  systems.  Therefore, 
the  DMO  simulation  environment  was  not  constant  for  all  users.  Some  examples  of 
upgrades/changes  to  the  environment  during  the  33-month  data  collection  period  included  (but 
are  not  limited  to): 

•  Upgrading  the  visual  databases  in  cockpits  #3  and  #4  to  use  the  same  photo-specific 
database  used  in  cockpits  #1  and  #2, 

•  upgrading  to  eight  visual  channels, 

•  upgrading  the  radios, 

•  installing  SCU-5  Situation  Awareness  DataLink  (SADL)  software, 

•  installing  new  ALQ-2 13  radar  waming/electronic  countermeasure  panels  and  5100  power 
PC  boards, 

•  adding  smoke  trails  to  missile  fly-outs, 

•  upgrading  the  brief/debrief  facility  with  Portable  Flight  Planning  Software  version  3.2, 

•  and  a  sixth  50-inch  plasma  debrief  display  for  AWACS. 
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Unfortunately,  most  of  these  upgrades  were  not  dated,  so  partitioning  the  opinion  datasets 
accordingly  was  not  feasible.  To  examine  if  changes  in  user  opinions  did  occur  over  time, 
we  performed  additional  analyses  by  breaking  up  the  3  3 -month  data  collection  period  into  six 
time  “slices.” 


DMO  Reaction  Survey  (Appendix  A): 

The  DMO  Reaction  survey  consisted  of  two  sections.  The  first  section  contained  58  “core” 
ratings  with  a  4-point  scale  (1  =  Strongly  Disagree;  2  =  Somewhat  Disagree;  3  =  Somewhat 
Agree;  and  4  =  Strongly  Agree),  while  the  second  section  consisted  of  open-ended  questions. 

The  DMO  Reaction  survey  was  administered  to  participants  towards  the  end  of  their  DMO 
training  research.  For  inclusion  in  the  current  work,  we  examined  only  those  DMO  Reaction 
surveys  completed  by  participants  in  the  overall  within- simulator  effectiveness  study,  a 
consistent  five-day  LAN-only  DMO  curriculum  (Schreiber  &  Bennett,  2006).  Of  the  384  pilots 
in  the  overall  study,  327  completed  the  DMO  Reaction  survey.  An  additional  49  AWACS 
controllers  also  completed  the  DMO  Reaction  survey.  All  but  two  of  the  327  pilots  were  male, 
with  an  age  range  between  24  and  54  years  (mean  =  33.0).  The  pilots  averaged  1,681  flight 
hours  up  to  the  time  they  participated  in  DMO  at  Mesa,  and  an  average  of  1,039  of  the  1,681 
total  hours  were  F-16  hours.  AWACS  demographic  infonnation  is  available  for  45  of  the  49 
participants.  All  but  three  of  the  45  controllers  were  male,  with  an  age  range  between  24  and  41 
years  (mean  =  30.4)  The  DMO  Reaction  survey  participants  were  not  randomly  sampled;  they 
vied  for  posted  training  research  opportunities  at  the  Mesa  Research  Site.  As  such,  these 
participants  may,  or  even  likely,  be  more  positively  disposed  overall  towards  DMO  than  some  of 
their  peers  and  their  ratings/opinions  provided  on  the  DMO  Reaction  form  should  be  interpreted 
accordingly. 

Due  to  the  large  number  of  statement  ratings,  we  desired  to  categorize  them  for  summary 
presentation.  Retired  F-16  SMEs  completed  a  categorization  task.  During  the  initial  phase,  three 
SMEs  independently  sorted  all  statements  into  categories,  with  each  SME  producing  between  six 
and  eight  categories.  From  these  22  potential  categories,  two  SMEs  then  collectively  assigned 
each  statement  to  a  category,  ultimately  deciding  that  only  seven  categories  were  required  to 
encompass  all  58  statements.  The  resulting  seven  summary  categories  are  shown  in  Table  2. 


Table  2  DMO  Reaction  Survey  Summary  Categories 


Category 

Home  Unit  Conditions 
Overall  Training  Value 
Syllabus/Mission  Flow 
DMO  Opinions 

General  Comments  About  the  DMO 
Environment 

DMO  Scenario  Characteristics 
DMO  Expectations 


Number  of  Statements 
6 

20 

8 

11 

4 

3 
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From  the  second  section  of  the  survey — the  open-ended  questions — we  did  not  choose  to  analyze 
many  of  the  responses,  as  most  open-ended  questions  were  geared  for  internal  developmental 
purposes  to  quickly  identify  and  fix  emerging  problem  areas.  As  such,  we  analyzed  only  the 
responses  to  the  first  open-ended  question  using  a  simple  content  analysis  approach. 

Specifically,  we  examined  the  participants’  responses  to: 

“List  the  top  five  things  you  feel  were  beneficial  about  the  training  you  received  here  at 
DMO.  Next  to  each  item,  please  state  why  it  was  beneficial.  ’’ 

The  318  pilots  and  43  air  weapons  controllers  provided  one  or  more  statements.  In  order  to 
analyze  these  statements,  segments  (pieces  of  text)  were  created.  Segments  consisted  of 
combining  one  “what  they  like”  statement  with  the  related  “why”  statement  (e.g.,  “Realism  of 
sims...  Limited  frustration  with  sim-isms”).  There  were  1,123  pilot  segments  and  187  AWACS 
segments.  Three  assistants  then  independently  sorted  and  categorized  the  segments,  with  each 
person  creating  between  13  and  16  categories.  After  completing  this  first  categorization  sorting 
task,  a  researcher  and  a  SME  then  used  these  initial  categories  to  jointly  finalize  a  single  set  of 
sixteen  categories  used  for  descriptive  analysis. 

MEC  Experience  Survey  (Appendix  B); 

To  identify  the  important  training  experiences,  the  final  critical  experiences  from  the  MEC 
process  were  utilized.  Since  the  majority  of  participants  in  the  current  work  flew  air-to-air 
oriented  missions,  we  chose  the  45  air  superiority  MEC  experiences  to  implement  in  a  survey  to 
capture  warfighter  opinion  on  how  well  the  Mesa  DMO  research  system  might  provide  each 
experience  (i.e.,  assess  perceived  training  utility).  For  comparative  purposes,  we  asked  each 
participant  to  rate  the  extent  to  which  they  felt  each  of  the  45  experiences  could  be  gained  in 
eight  different  environments  (360  total  ratings),  specifically: 

1.  Ready  Aircrew  Program  (RAP)  flying  events— all  except  Flag  and  Composite  Force  Training 
(CFTR) 

2.  RAP  (Flag  and  CFTR  events) 

3.  MTC/FMT  (Mission  Training  Center/Full  Mission  Trainer) 

4.  Operation  Northern  Watch  (ONW),  Operation  Southern  Watch  (OSW) 

5.  UTD  (Unit  Training  Device) 

6.  Mesa  DMO 

7.  WTT/DTT  (Weapons  and  Tactics  Trainer/Desktop  Trainer) 

8.  Sustained  Combat  Ops  (e.g.,  Operation  Allied  Force  (OAF),  Desert  Stonn) 

The  MEC  experience  by  environment  survey  was  administered  to  32  operational  pilots  who 
participated  in  five-day  Mesa  training  research.  All  of  the  participants  were  male  with  an  age 
range  between  26  and  44  years  (mean  =  31.64).  The  pilots  averaged  1,388  flight  hours,  of  which 
1,072  hours  were  F-16  hours.  All  32  pilots  completed  the  survey.  As  with  the  DMO  Reaction 
Survey,  the  pilots  who  completed  the  MEC  experience  survey  were  not  randomly  sampled. 
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RESULTS 

DMO  Reaction  Survey:  Rated  Statements 


Before  proceeding  with  the  rated  statement  analysis,  we  needed  to  recode  10  of  the  58  statements 
to  maintain  consistency  across  all  rated  items  that  a  higher  numbered  score  reflected  a  favorable 
DMO  rating  score  (i.e.,  only  for  those  recoded  items,  a  “4”  rating,  for  example,  was  recoded  into 
“Strongly  Disagree,”  while  for  the  non-recoded  items  a  “4”  rating  remained  “Strongly  Agree”). 

In  Tables  3-9,  the  recoded  items  are  denoted  with  an  asterisk  (*).  The  rated  statements  were 
grouped  and  are  reported  here  according  to  their  summary  categories  from  Table  2. 

Prior  to  analyzing  the  overall  aggregated  dataset,  we  chose  to  partition  the  pilot  reaction  rating 
database  into  six  time  “slices”  (about  six  months  each)  to  investigate  whether  or  not  user 
opinions  changed  over  time  as  a  function  of  the  aforementioned  changes  in  the  DMO 
environment.  While  there  were  insufficient  AWACS  operators  to  “time-slice”  the  database,  six 
time  slices  provided  sufficient  pilot  rating  sample  sizes  for  this  investigation  (between  43  and  64 
pilots  per  time  slice).  Results  from  these  analyses  revealed  a  few  statistically  significant 
differences  in  user  opinion  summary  category  ratings  over  time.  Only  one  of  the  seven  summary 
categories — Scenario  Characteristics— revealed  a  significant  linear  trend  at  an  alpha  criterion  of 
.05,  and  it  was  in  the  expected  upwards  direction.  Only  the  DMO  Expectations  category 
revealed  a  significant  quadratic  trend  at  the  .05  level,  but  four  of  the  seven  categories  revealed 
significant  cubic  trends.  Despite  some  of  these  significant  time  results,  we  nonetheless  chose  to 
aggregate  the  database  over  time  and  report  here  the  results  for  the  entire  time  period  for  which 
data  were  collected.  We  chose  this  approach  primarily  because  (a)  though  some  results  were 
statistically  significant,  we  found  the  tight  range  in  observed  ratings  over  time  to  be  practically 
insignificant,  and  (b)  stable,  consistent,  and  meaningful  time-based  trends  from  this  tight  results 
range  are  simply  not  apparent.  Graphs  and  detailed  results  of  the  time-based  analyses  are 
provided  in  Appendix  C. 

For  the  entire  33-month  data  collection  period,  the  results  for  the  individual  statements  and 
summary  scores  for  both  pilots  and  AWACS  participants  are  shown  in  Tables  3-9.  For  the  327 
F-16  pilots  across  all  individually  rated  statements,  all  58  of  the  average  rated  scores  were  rather 
favorable;  their  average  scores  for  a  given  statement  ranged  from  a  low  of  2.56  (“As  a  result  of 
this  training,  I  have  improved  my  VID  tactics”)  to  a  high  of  3.94  for  two  statements  (“I  would 
recommend  this  training  experience  to  other  pilots/controllers”  and  “DMO  will  positively  impact 
my  combat  mission  readiness”),  with  only  four  of  the  58  individual  statements  receiving  an 
average  rating  below  a  three.  Across  the  seven  summary  categories,  all  the  summary  category 
weighted  means  were  found  to  be  above  3.0  (3.20-3.72). 

As  with  the  pilot  scores,  for  the  49  AWACS  controllers  across  all  individually  rated  statements, 
the  average  rated  scores  for  the  58  statements  were  fairly  high  in  general.  Their  average  scores 
for  a  given  statement  ranged  from  a  low  of  2.45  (“This  training  provided  excellent  experience  in 
radar  mechanics”)  to  a  very  high,  almost  unanimous  score  of  3.98  (“I  would  recommend  this 
training  experience  to  other  pilots/controllers”).  Fess  than  20%  (9/58)  of  all  the  individual 
statements  received  an  average  rating  below  a  three.  Across  the  seven  summary  categories,  all 
the  summary  weighted  means  for  AWACS  participants  were  found  to  be  above  3.0  (3.01-3.69). 
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Table  3  DMO  Expectations  Ratings. 


DMO  Reaction:  DMO  Expectations 

Pilots 

AWACS 

Statement 

Mean  (n;  s.e.) 

Mean  (n; 

s.e.) 

I  came  to  this  training  with  low  expectations* 

3.54  (326;  .04) 

3.40  (47; 

■12) 

I  was  prepared  for  the  type  of  experience  I  observed  / 

received  in  DMO 

3.36  (327;  .04) 

3.51  (47; 

■11) 

I  expected  more  from  this  training  than  was  delivered* 

3.66  (327;  .03) 

3.51  (47; 

.13) 

I  expected  this  training  to  be  a  valuable  experience 

3.78  (327;  .03) 

3.72  (47; 

.08) 

The  goals  I  listed  for  this  DMO  exercise  were  met 

3.68  (321;  .03) 

3.64  (47; 

.09) 

The  expectations  I  listed  for  this  DMO  exercise  were  met 

3.75  (321;  .03) 

3.77  (44; 

.06) 

DMO  Expectations  Overall  Score  (N=1949) 

3.63  (1949;  .03) 

3.59  (279 

.10) 

Table  4  DMO  Opinions  Ratings 

• 

DMO  Reaction:  DMO  Opinions 

Pilots 

AWACS 

Statement 

Mean  (n;  s.e.) 

Mean  (n; 

s.e.) 

This  training  experience  has  motivated  me  to  seek  similar 

training  opportunities 

3.85  (327;  .02) 

3.83  (47; 

.06) 

The  DMO  learning  objectives  were  presented  clearly 

3.50  (327;  .04) 

3.26  (47; 

■ID 

I  would  recommend  this  training  experience  to  other  pilots  / 

controllers 

3.94  (326;  .02) 

3.98  (48; 

.02) 

DMO  offers  an  excellent  opportunity  to  view  multiple 

complex  presentations 

3.88  (327;  .03) 

3.81  (48; 

.08) 

DMO  provides  realistic  training  and  experience 

3.54  (327;  .03) 

3.22  (48; 

■ID 

DMO  has  limited  military  training  value* 

3.76  (327;  .04) 

3.56  (48; 

.13) 

Using  DMO  in  conjunction  with  live  flying  is  worthwhile 

3.81  (325;  .03) 

3.92  (48; 

.04) 

I  would  like  to  see  DMO  operational  applications  expanded 

3.81  (327;  .02) 

3.96  (49; 

.03) 

DMO  provides  valuable  experience  in  combat  mission  tactics 

3.89  (327;  .02) 

3.90  (49; 

.04) 

DMO  should  be  a  part  of  all  future  spin-up  exercises 

3.70  (325;  .03) 

3.77  (47; 

.08) 

The  DMO  missions  I  experienced  were  unrealistic* 

3.27  (326;  .05) 

3.35  (48; 

■11) 

Overall  DMO  Opinions 

3.72  (3591;  .03) 

3.69  (527 

.07) 

Table  5  DMO  Scenario  Characteristics. 


DVIO  Reaction:  DMO  Scenario  Characteristics  Pilots  AWACS 


Statement _ Mean  (n;  s.e.)  Mean  (n;  s.e.) 


The  scenarios  were  well  designed 

3.67 

(327;  .03) 

3.63 

(49; 

.08) 

The  threats  in  DMO  behaved  as  they  do  in  the  operational 

environment 

3.0 

(325; .04) 

3.09 

(46; 

•11) 

Rate  the  degree  to  which  you  feel  the  Intel  briefing  was 

useful  for  missions  flown 

3.03 

(301;  .04) 

2.98 

(42; 

.10) 

Overall  DMO  Scenario  Characteristics 

3.23 

(953; .04) 

3.23  i 

(137; 

.10) 
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Table  6  General  Comments  about  the  DMO  Environment. 


DMO  Reaction:  General  Comments  about  the  DMO 
Environment 

Pilots 

AWACS 

Statement 

Mean  (n;  s.e.) 

Mean  (n;  s.e.) 

The  visual  scenes  in  DMO  were  realistic 

3.17(327;  .04) 

2.86  (42;.  14) 

DMO  missions  and  engagements  accurately  represent  how 
things  happen  in  the  operational  world 

2.91  (327;  .04) 

3.0(48;  .11) 

The  visual  databases  were  realistic 

3.23  (323;  .04) 

2.91  (44;  .12) 

The  fidelity  of  the  DMO  environment  was  sufficient  to 
achieve  the  desired  mission/training  objectives 

Overall  General  Comments  About  the  DMO 

3.50  (327;  .04) 

3.28  (47;  .13) 

Environment 

3.20  (1304;  .04) 

3.01  (181;  .12) 

Table  7  Home  Unit  Conditions. 

DMO  Reaction:  Home  Unit  Conditions 

Pilots 

AWACS 

Statement 

Mean  (n;  s.e.) 

Mean  (n;  s.e.) 

I  will  have  an  opportunity  to  use  what  I’ve  learned  at  my 
operational  unit 

Daily  operational  unit  requirements  are  conducive  to  my 

3.80  (327;  .03) 

3.83  (47;  .06) 

maintaining  the  skills  I  have  learned  in  DMO 

I  can  routinely  get  the  type  of  experience  I  had  in  DMO  at 

2.94  (324;  .06) 

3.09  (46;  .14) 

my  home  unit* 

3.79  (327;  .03) 

3.35  (48;  .13) 

At  my  unit,  I  routinely  get  to  fly  against  realistic  threats* 

3.29  (327;  .05) 

2.74  (43;  .15) 

There  are  a  sufficient  number  of  4vX  flying  opportunities  at 
my  unit* 

3.29  (327;  .04) 

2.98  (47;  .13) 

I  routinely  get  to  practice  tactics  in  multiple  aircraft 
environments  at  my  operational  unit* 

3.28  (327;  .05) 

2.68  (44;.  15) 

Overall  Home  Unit  Conditions 

3.40  (1959;  .04) 

3.01  (275;  .13) 

Table  8  Overall  Training  Value. 

DMO  Reaction:  Overall  Training  Value 

Pilots 

AWACS 

Statement 

Mean  (n;  s.e.) 

Mean  (n;  s.e.) 

DMO  will  positively  impact  my  combat  mission  readiness 
Participating  in  the  DMO  program  was  a  good  use  of  my 

3.94  (326;  .01) 

3.85  (48;  .06) 

time 

3.93  (327;  .02) 

3.96  (47;  .03) 

The  material  I  learned  in  DMO  is  useful  for  my  future  flying 
/  AWACS  experience 

3.88  (326;  .02) 

3.91  (47;  .07) 

The  missions  helped  me  to  effectively  meet  my  learning 
objectives 

This  training  provided  excellent  experience  in  radar 

3.80  (326;  .03) 

3.75  (48;  .08) 

mechanics 

3.83  (327;  .03) 

2.45  (40;  .16) 
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Table  9  Overall  Training  Value.  (Cont’d) 


DMO  Reaction:  Overall  Training  Value 

Statement 

As  a  result  of  this  training,  I  have  improved  my  mission 
planning  skills 

I  have  improved  my  tactical  skills  as  a  result  of  the  DMO 
experience 

DMO  has  considerable  combat  mission  training  value 
I  am  confident  that  I  can  perform  the  tasks  taught  in  this 
training  program  in  the  field 

DMO  improved  my  understanding  of  critical  combat  skills 
I  improved  my  briefing  and  debriefing  skills 
DMO  has  improved  my  combat  mission  readiness 
This  training  provided  excellent  practice  on  3-1 
communication 

As  a  result  of  this  training,  I  have  improved  my  BVR  tactics 
I  feel  I  will  be  better  prepared  for  to  brief  and  lead  a  mission 
due  to  this  training 

I  am  confident  I  will  perform  better  as  a  fighter  pilot  / 
controller  as  a  result  of  DMO 

As  a  result  of  this  training,  I  have  improved  my  VID  tactics 
DMO  has  helped  me  improve  my  team  coordination  skills 
This  training  helped  me  improve  my  combat  situational 
awareness 

The  skills  I  trained  in  DMO  are  the  same  as  those  I  am 
expected  to  perfonn  in  the  field 

Overall  Training 


Pilots 

Mean  (n;  s.e.) 

AWACS 
Mean  (n;  s.e.) 

3.23  (327;  .04) 

3.26  (46;  .13) 

3.89  (327;  .02) 

3.90  (327;  .02) 

3.92  (48;  .04) 
3.83  (48;  .07) 

3.68  (327;  .03) 
3.68  (326;  .03) 
3.31  (319;  .04) 
3.85  (326;  .02) 

3.67  (48;  .09) 

3.76  (49;  .06) 

2.76  (45;  .13) 
3.82  (49;  .06) 

3.92  (326;  .02) 

3.93  (327;  .01) 

3.94  (49;  .03) 
3.64  (39;  .11) 

3.80  (324;  .03) 

3.38  (42;  .13) 

3.88  (327;  .02) 
2.56(327;  .05) 
3.62  (327;  .03) 

3.83  (48;  .07) 
2.71  (28;  .18) 
3.60  (48;  .08) 

3.79  (327;  .03) 

3.82  (49;  .06) 

3.39  (325;  .04) 

3.69  (6521;  .03) 

3.65  (49;  .09) 

3.58  (915;  .09) 

Table  10  Syllabus  Mission  Flow. 


DMO  Reaction:  Syllabus  Mission  Flow 

Pilots 

AWACS 

Statement 

Mean  (n;  s.e.) 

Mean  (n; 

s.e.) 

The  missions  were  sequenced  in  a  way  that  facilitated  my 

learning 

3.68  (327;  .03) 

3.75  (48; 

.07) 

I  felt  the  training  was  appropriately  paced 

3.73  (327;  .03) 

3.85  (47; 

.06) 

The  missions  were  challenging 

3.86  (327;  .02) 

3.64  (47; 

.09) 

It  was  difficult  to  keep  up  with  the  pace  of  the  missions* 

2.93  (326;  .05) 

3.30  (47; 

.10) 

The  missions  were  not  very  realistic* 

3.21  (326;  .05) 

3.0(47; 

.12) 

In  training,  I  was  able  to  apply  lessons  learned  from  previous 

missions  to  new  ones 

3.76  (327;  .03) 

3.83  (47; 

.06) 

The  syllabus  and  missions  were  presented  in  an  organized 

manner 

3.59(326;  .03) 

3.51  (47; 

.09) 

The  missions  and  engagements  were  very  challenging 

3.79  (326;  .03) 

3.38  (48; 

.10) 

Overall  Syllabus  Mission  Flow 

3.57  (2612;  .03) 

3.53  (378: 

;  .09) 
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DMO  Reaction  Survey:  Open-Ended  Responses 


Before  calculating  item  frequencies,  we  checked  the  reliability  of  the  coded  segments  using  80% 
agreement  as  our  criterion  (Lombard,  Snyder-Duch,  &  Bracken,  2005;  Neuendorf,  2002).  This 
agreement  percentage  was  calculated  using  the  total  number  of  identical  coded  statements  (ICS) 
divided  by  NSC,  or  the  total  number  of  segments  categorized  (i.e.,  ICS/NCS).  The  coding  for 
the  pilots’  data  achieved  a  91.6%  agreement  and  the  data  from  the  AWACS  achieved  an  80.7% 
agreement.  The  number  of  statements  per  category  is  provided  for  both  the  pilots  and  the 
AWACS  participants  in  Table  10  and  the  percentages  of  total  responses  are  graphed  in  Figure  5. 
Results  varied  by  pilots  and  AWACS  participants.  With  a  combined  total  of  over  half  their 
comments,  pilots  most  liked  the  realistic  qualities  or  the  skill  improvement/acquisition.  AWACS 
operators,  on  the  other  hand,  responded  with  a  combined  total  of  over  40%  of  their  comments 
that  they  most  liked  the  scenarios  and  the  skill  improvement  gained  from  the  briefs/debriefs. 

Table  11  Number  of  comments  by  category. 


Category 

%  of 

comments 

(pilots) 

%  of 

comments 

(AWACS) 

Frequency 

(pilots) 

Frequency 

(AWACS) 

Realistic  Qualities 

28.14% 

1.07% 

316 

2 

Skill  Improvement/Acquisition 

24.49% 

2.14% 

275 

4 

Briefs/Debriefs  (Training  specific 
facilities) 

10.95% 

8.02% 

123 

15 

Communication 

7.30% 

8.56% 

82 

16 

Tactics 

6.77% 

7.49% 

76 

14 

Scenarios  (Quantity/V ariety/Quality) 

3.83% 

22.99% 

43 

43 

Controller/ AWACS  Integration 

3.21% 

8.02% 

36 

15 

SIM  Characteristics 

3.21% 

7.49% 

36 

14 

Situation  Awareness 

3.03% 

6.99% 

34 

13 

Cold  Ops 

2.32% 

2.67% 

26 

5 

Threats 

2.23% 

2.14% 

25 

4 

Incidentals  (non-DMO  references) 

1.96% 

1.60% 

22 

3 

Other  Training  Related  Benefits 

.98% 

1.60% 

11 

3 

Weapons/Weapon  Employment 

.80% 

0% 

9 

0 

Briefs/Debriefs  (Skill  Improvement 
Acquisition) 

.53% 

18.72% 

6 

35 

Briefs/Debriefs  (Non-Specific) 

16% 

.53% 

3 

1 
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Percentage  of  Statements  Coded 


Pilot  and  AWACS  Top  DMO  Categories 


Figure  5  Percentage  of  overall  responses  by  category. 

As  a  final  examination  of  the  DMO  Reaction  Survey,  and  only  because  there  is  typically  fair 
interest,  we  report  here  ten  notable  quotations  provided  by  the  participants.  We  remind  the 
reader  that  these  are  selected  quotations  from  the  users;  these  quotations  are  not  reported  in  any 
scientific  manner  and  should  not  be  construed  as  representative  of  all  the  direct  quotations 
provided. 

1 .  We  were  able  to  get  training  we  can't  get  at  home  without  typical  distractions. 

2.  The  excellent  simulations  combined  with  outstanding  debriefing  tools  plus  the  amount  of 
"sorties"  in  such  a  short  time  led  to  increased  skills  in  all  areas 

3.  Equipment. .  .Love  the  Hi  tech  debrief  tools  (training  rules) 

4.  Execution. .  .Most  efficient  I  have  ever  seen 

5.  Facilities. .  .Top  notch  visual  /  realistic  trainer  cockpit 

6.  Only  place  to  receive  this  quantity  and  level  of  training 


15 


7.  Realism. .  .Train  how  we  fight 

8.  Multiple  bandits  with  realistic  contact  ranges  is  invaluable 

9.  Ground  control  intercept  (GCI)  and  fighter  interaction.  .  .  I  have  never  had  this  level  of 
understanding  of  the  importance  of  communication  pacing  and  how  critical  it  is  to  use  the 
correct  verbiage 

10.  Quality  of  Debriefs...  Getting  a  100%  accurate  picture  with  all  cockpit  data  is  an 
important  part  of  learning  from  the  debrief 

MEC  DMO  Experiences  Survey: 

The  pilots  rated  each  of  the  45  MEC  experiences  on  a  5-point  scale  as  to  the  extent  that  different 
environments  provide  training  for  that  experience.  For  reference,  the  scale  used  was: 

0  =  Not  at  all/Does  Not  Apply 

1  =  To  a  Slight  Extent 

2  =  To  a  Moderate  Extent 

3  =  To  a  Substantial  Extent 

4  =  To  a  Great  Extent 

Across  all  45  MEC  experiences,  the  average  ratings  per  environment  were  computed  and  a 
within-subjects  analysis  of  variance  (ANOVA)  performed.  The  differences  in  average  ratings 
between  environments  was  found  to  be  highly  significant,  F(7,217)  =  1 1.96,  p  <  .01,  with  the 
Mesa  DMO  environment  rated  highest  overall  and  the  WTT/DTT  enviromnent  rated  lowest 
overall.  Contrasts  tests  comparing  the  Mesa  DMO  enviromnent  average  against  the  average  of 
all  other  environments  revealed  that  Mesa  was  rated  significantly  higher  (at  alpha  =  .01)  than  all 
but  one  pairwise  comparison  (the  only  exception  being  the  RAP  Flag/CFTR  environment 
category).  Therefore,  it  was  not  surprising  to  find  that  the  distribution  of  ratings  varied  with 
environment,  as  shown  in  Table  11.  Ratings  for  the  WTT/DTT  and  ONW/OSW  environments 
were  generally  negative,  while  ratings  for  Mesa  DMO  were  most  positive.  Only  three  of  the 
eight  environments  (Mesa  DMO  and  the  two  RAP  environment  categories)  were  judged  to 
provide  half  or  more  of  the  MEC  experiences  “to  a  moderate  extent”  or  better. 

Individual  cell  rating  results  show  drastically  different  averages  for  each  of  the  eight 
environments  across  the  45  various  experiences  (i.e.,  the  360  cells),  ranging  from  the  lowest 
rating  of  just  .25  (tied)  for  two  environments  (MTC/FMT  and  WTT/DTT)  providing  the  “G- 
induced  physical  limitations”  experience,  to  a  high  of  3.66  for  the  Mesa  DMO  environment 
providing  the  “1:3+  Force  Ratio”  experience.  For  each  of  the  45  MEC  experiences,  the  highest 
and  lowest  rated  environment  was  identified  and  tabulated;  the  results  are  shown  in  Table  12. 

The  Mesa  DMO  environment  was  rated  as  best  (or  tied  as  best)  for  providing  29/45  (64.4%)  of 
the  MEC  experiences,  while  the  WTT/DTT  environment  category  was  rated  worst  (or  tied  for 
worst)  for  33/45  (73.3%)  of  the  MEC  experiences.  The  complete  average  rating  results  per  cell 
(i.e.,  each  experience  by  each  environment)  are  provided  as  referenced  in  Appendix  D. 
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Table  12  Ratings  averaged  over  all  45  MEC  experiences. 


Environment 

Avg  rating  over 
all  45 

experiences 

%  of 

experiences 
rated  3  or 
higher 

%  of 

experiences 
rated  2  or 
higher 

%  of  experiences 
rated  1  or  higher 

Mesa  DMO 

2.65 

40% 

84.4% 

95.6% 

RAP 

Flag/CFTR 

2.37 

4.4% 

75.6% 

100% 

RAP  except 
Flag/CFTR 

2.09 

4.4% 

60% 

97.8% 

UTD 

1.85 

0 

44.4% 

93.3% 

Sustained 
Combat  Ops 

1.74 

2.2% 

33.3% 

91.1% 

MTC/FMT 

1.54 

0 

11.1% 

86.7% 

ONW/OSW 

1.08 

0 

0 

60% 

WTT/DTT 

0.93 

0 

0 

40% 

Table  13  Number  and  percentage  of  experiences  rated  highest  or  lowest  for  each  environment. 


Environment 

#  Experiences 
rated  highest  (or 
tied  for  highest) 

% 

#  Experiences 
rated  lowest  (or 
tied  for  lowest) 

% 

All  RAP  except 
Flag/CFTR 

3 

6.7% 

0 

0 

RAP  Flag/CFTR 

9 

20% 

0 

0 

MTC/FMT 

0 

0 

3 

6.7% 

ONW/OSW 

0 

0 

14 

31.1% 

UTD 

2 

4.4% 

0 

0 

Mesa  DMO 

29 

64.4% 

0 

0 

WTT/DTT 

0 

0 

33 

73.3% 

Sustained  Combat 
Ops 

4 

8.9% 

0 

0 

DISCUSSION 


Upon  reviewing  the  results,  overall  validity  of  the  user  responses  seems  intact,  thereby  allowing 
us  to  draw  some  meaningful  conclusions.  If  the  participants  were  not  taking  adequate  time  to 
complete  the  DMO  Reaction  survey  or  were  not  taking  the  survey  seriously,  we  would  expect 
conflicting  ratings  for  similarly  rated  statement  elements.  Some  elements  within  the  DMO 
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Reaction  survey  were  specifically  worded  similarly  for  this  evaluation,  and  neither  pilots  nor 
AWACS  participants  provided  conflicting  ratings  (e.g.,  ratings  for  “I  expected  this  training  to  be 
a  valuable  experience”  were  high  AND  ratings  for  “I  expected  more  from  this  training  than  was 
delivered”  were  low  [before  recoding]).  As  another  example,  from  the  experience  by 
environments  survey  certain  key  cells  showed  distinctly  different  (and  expected)  results  (e.g., 
low  ratings  for  “G-induced  physical  limitations”  should  be  observed  in  a  fixed-base  simulation 
environment).  Furthermore,  the  training  syllabi  were  geared  more  towards  pilots,  so  higher 
overall  pilot  DMO  Reaction  survey  ratings  compared  to  AWACS  was  to  be  expected.  Given  the 
valid  responses,  we  now  discuss  the  results  from  each  analysis  in  turn. 

Overall,  the  DMO  Reaction  rating  data  shows  acceptance  of  the  Mesa  research  environment,  as 
clearly  evidenced  by  the  fact  that  scores  for  all  participants,  both  pilots  and  AWACS  controllers, 
were  relatively  high  for  all  seven  summary  categories.  Of  all  the  rated  elements  on  the  reaction 
form,  statements  recommending  the  experience  to  peers  arguably  may  be  the  most  indicative  of 
user  opinion  regarding  the  environment.  The  single  most  highly  rated  statement  (of  all  58  rated 
statements)  for  both  pilots  and  AWACS  controllers  was  the  same,  was  very  nearly  unanimous, 
and  was  of  the  peer  recommendation  nature:  “I  would  recommend  this  training  experience  to 
other  pilots/controllers”  was  rated  by  all  but  one  of  49  controllers  and  all  but  16  of  327  pilots 
with  the  highest  rating  possible  of  “Strongly  Agree.” 

Over  the  course  of  the  data  collection  period,  continuous  improvements  were  made  to  the  DMO 
environment,  but  we  found  only  small  (and  few  significant)  changes  in  DMO  Reaction  ratings 
over  time.  We  expected  a  more  noticeable  rise  in  average  observed  DMO  Reaction  ratings. 
Outside  of  a  likely  cause  for  raters  tending  to  anchor,  one  other  possible  explanation  for  the 
relatively  flat  result  is  that,  as  pilots  have  become  more  familiar  and  comfortable  with  DMO 
systems  over  the  past  few  years,  the  expectations  may  have  also  raised.  Expectation  increases 
could  easily  be  negating  or  “canceling  out”  corresponding  increases  in  user  perceptions,  thereby 
yielding  roughly  the  same  ratings  over  time. 

We  must  preface  our  discussion  of  the  DMO  Reaction  content  analysis  results  by  stating  that  the 
question  analyzed  was  leading,  asking  only  for  the  perceived  “likes”  of  the  environment.  As 
such,  those  statements  infrequently  reported  should  certainly  not  be  interpreted  as  “dislikes” 
about  the  environment.  Additionally,  an  opinionated  and  verbose  participant  would  provide  up 
to  five  statements,  while  some  other  participants  may  have  provided  only  one  (or  even  none). 
Therefore,  participants  and  their  statements  are  disproportionately  represented  in  the  data.  With 
that  being  said,  we  believe  that  the  content  analysis  still  does  reveal  some  interesting  patterns. 

As  could  be  expected  with  different  backgrounds  and  vocations,  the  pilots  and  AWACS 
personnel  provided  statements  that  generally  differed  in  what  they  best  liked  about  the 
environment.  However,  one  common  perception  shared  by  both  demographics  was  the  skill 
acquisition.  But,  the  manner  in  which  they  reported  it  differed;  the  AWACS  operators  reported 
the  skill  acquisition  primarily  as  a  function  of  brief/debrief,  while  the  pilots  reported  it  in  more  of 
a  general  context  (with  some  trepidation,  we  infer  that  a  fair  portion  of  the  skill  acquisition  is 
therefore  due  to  the  simulator  “flying”).  These  self-generated  user  statements  regarding  skill 
acquisition  are  extremely  well  buttressed  by  our  other  research  documenting  the  extent  of  in¬ 
simulator  DMO  learning  at  the  Mesa  Research  Site  (Schreiber  &  Bennett,  2006;  Schreiber, 

Stock,  &  Bennett,  2006;  Schreiber,  Gehr,  &  Bennett,  2006). 
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The  experience  ratings  revealed  that  the  pilots  clearly  believe  the  Mesa  DMO  environment 
affords  substantial  training  utility  in  providing  the  various  MEC  experiences,  as  evidenced  by: 

•  Mesa  DMO  had  the  highest  overall  average  rating  across  all  experiences, 

•  the  Mesa  DMO  environment  was  rated  as  best  providing  29/45  of  the  experiences,  and 

•  Mesa  DMO  was  rated  as  able  to  provide  84.4%  of  all  the  experiences  “to  a  moderate 
extent”  or  better 

— all  three  statistics  were  higher  for  Mesa  DMO  than  all  seven  other  environments. 

Though  the  average  overall  ratings  were  highest  for  the  Mesa  DMO  environment,  care  should 
still  be  taken  when  comparing  overall  ratings  among  environments,  especially  Mesa  versus  the 
other  environments.  First,  and  probably  of  least  impact  on  the  results,  these  results  assume  each 
experience  is  equal — an  unlikely  assumption.  If  the  experiences  were  weighted  by  importance, 
we  might  have  observed  slightly  different  results.  Second,  as  discussed  in  the  methods  section, 
pilots  were  not  randomly  selected  and  may  be  more  positively  disposed  towards  DMO  (though  if 
this  were  a  strong  effect,  we  would  have  expected  somewhat  higher  results  for  the  MTC/FMT 
environment).  And  lastly,  most  expenses  for  participation  at  Mesa  were  covered  by  Mesa, 
possibly  influencing  ratings.  Though  some  of  these  caveats  may  diminish  the  interpretive 
strength  of  the  Mesa  DMO  MEC  experience  ratings,  we  believe  it  does  so  only  “to  a  slight 
extent”  (especially  since  we  believe  fighter  pilots  tend  to  give  their  honest  opinions  on 
anonymous  surveys).  On  the  other  hand,  higher  ratings  for  many  of  the  MEC  experiences  were 
expected  of  the  Mesa  DMO  site  due  to  the  nature  of  the  training  research  performed  there.  The 
scenarios  and  syllabi  used  at  Mesa  during  this  study  were  carefully  designed  to  train  specific  air- 
to-air  MEC  skills,  so  it  logically  follows  that  ratings  of  the  environment  could  trend  higher  as  a 
result.  How  much  each  of  these  factors  influenced  ratings  is  unknown,  but  putting  these 
scientific  philosophical  interpretive  moderators  aside,  we  believe  it  is  still  clear  that  the 
warfighters  certainly  do  not  view  the  Mesa  DMO  training  utility  as  negative,  and  they  likely 
perceive  it  as  one  of  the  more  utilitarian  environments  in  which  to  gain  the  MEC  experiences. 

Interestingly,  some  of  the  perceived  experience  ratings  were  likely  low  because  pilots  did  not 
attempt  to  extrapolate  beyond  what  was  personally  experienced.  Or,  many  pilots  used  the  “Does 
Not  Apply”  rating  of  zero  because  they  were  unaware  that  a  capability  existed.  For  example,  the 
experience  “air  refueling”  was  rated  quite  low  in  both  the  Mesa  DMO  and  the  MTC/FMT 
environments.  Air  refueling  was  never  experienced  during  the  course  of  any  of  the  Mesa  DMO 
training  research  weeks,  yet  the  Mesa  DMO  environment  could  easily  be  configured  to  provide 
that  type  of  experience  and  the  F-16  MTC  (at  Shaw  AFB)  can  currently  provide  air  refueling 
scenarios  (but  air  refuelings  are  rarely  done  in  the  MTC  environment). 
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AMRAAM 
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ATES 
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DTT 
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FMT 
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LAN 

MEC 

MiG 

Mk 

MTC 

OCA 

ONW/OSW 

RAP 

SADL 

scu 

SME 

USAF 

UTD 

VID 

WAN 

WTT/DTT 


Air-to-Air 

Antiaircraft  Artillery 
Air  Force  Base 

Air  Force  Research  Laboratory 

AFRL  designation  for  Mesa  Research  Site 

Air  Intercept  Missile 

Advanced  Medium  Range  Air-to-Air  Missile 
Analysis  of  Variance 
Automated  Threat  Engagement  System 
Airborne  Warning  and  Control  System 
Composite  Force  Training 
Distributed  Interactive  Simulation 
Distributed  Mission  Operations 
Desktop  Trainer 

Institute  of  Electrical  and  Electronics  Engineers 

Full  Mission  Trainer 

Ground  Control  Intercept 

Instructor  Operator  Station 

Local  Area  Network 

Mission  Essential  Competencies 

Initials  of  Mikoyan-Gurevich  Design  Bureau 

Mark 

Mission  Training  Center 
Offensive  Counter  Air 

Operations  Northern  Watch/  Operations  Southern  Watch 

Ready  Aircrew  Program 

Situation  Awareness  DataLink 

Software  Capabilities  Upgrade 

Subject  Matter  Expert 

United  States  Air  Force 

Unit  Training  Device 

Virtual  Image  Display 

Wide  Area  Network 

Weapon  and  Tactics  Trainer/ Desk  Top  Trainer 
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APPENDIX  A 
DMO  Reaction  Survey 


Reactions  to  DMO/DMT  and  Simulation-based  Training  Experience 

I  UNDERSTAND  THAT  MY  PARTICIPATION  IN  THIS  RESEARCH  EFFORT  IS  VOLUNTARY  AND  THAT 
MY  COMPLETION  OF  THIS  INSTRUMENT  AND/OR  MY  PROVIDING  DATA  TO  AFRL/HEA 
RESEARCHERS  CONSTITUTES  MY  CONSENT  TO  PARTICIPATE  IN  THE  RESEARCH  EFFORT. 

The  next  few  pages  contain  statements  to  assess  how  you  feel  about  the  concept  of  Distributed 

Mission  Training  (DMT)  as  demonstrated  in  the  training  syllabus  you  have  participated  in  this 

week.  Although  you  may  have  experienced,  seen  or  heard  about  other  similar  programs  in  the 

field  or  proposed,  we  are  specifically  interested  in  your  AFRL/HEA  experience.  Please  read 

each  statement  carefully  and  decide  which  of  the  response  choices  is  most  true  about  your 

reactions  to  AFRL/HEA  DMT  and  to  the  training  program.  Please  make  your  responses  by 

circling  the  number  that  indicates  how  strongly  you  agree  or  disagree  with  each  statement.  A 

rating  of  “4"  means  you  strongly  agree  with  the  statement  and  a  rating  of  “1”  means  you 

strongly  disagree  with  the  statement. 

On  the  last  page  are  a  few  short  answer  questions  that  will  help  us  improve  specific  training- 
related  areas  for  the  future. 

Date: _  Unit: _  5  Digit  ID  Number: _ 

Syllabus:  (circle  one) 

MQT 

FLUG 

IPUG 

WIC 

ADV.  AIR-to- AIRAA- 1  CT  AA-2A  CT 
AA-2G  CT 
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Strongly  Agree 


Somewhat  Agree 


Somewhat  Disagree 


Strongly  Disagree 

I  will  have  an  opportunity  to  use  what  I’ve  learned  at  my  operational  unit. 

1 

DMO  will  positively  impact  my  combat  mission  readiness. 

1 

This  training  experience  has  motivated  me  to  seek  similar  training  opportunities. 

1 

Daily  operational  unit  requirements  are  conducive  to  my  maintaining  the  skills  I  have  learned 
in  DMO. 

1 

Participating  in  the  DMO  program  was  a  good  use  of  my  time. 

1 

The  DMO  learning  objectives  were  presented  clearly. 

1 

The  missions  were  sequenced  in  a  way  that  facilitated  my  learning. 

1 

The  visual  scenes  in  DMO  were  realistic. 

1 

The  material  I  learned  in  DMO  is  useful  for  my  future  flying  /  AW  ACS  experience. 

1 

The  missions  helped  me  to  effectively  meet  my  learning  objectives. 

1 

I  came  to  this  training  with  low  expectations. 

1 

I  would  recommend  this  training  experience  to  other  pilots  /  controllers. 

1 

I  felt  the  training  was  appropriately  paced. 

1 

I  was  prepared  for  the  type  of  experience  I  observed  /  received  in  DMO. 

1 

The  missions  were  challenging. 

1 

It  was  difficult  to  keep  up  with  the  pace  of  the  missions. 

1 

The  missions  were  not  very  realistic. 

1 

DMO  offers  an  excellent  opportunity  to  view  multiple  complex  presentations. 

1 

I  can  routinely  get  the  type  of  experience  I  had  in  DMO  at  my  home  unit. 

1 

In  training,  I  was  able  to  apply  lessons  learned  from  previous  missions  to  new  ones. 

1 

DMO  provides  realistic  training  and  experience. 

1 

This  training  provided  excellent  experience  in  radar  mechanics. 

1 

I  expected  more  from  this  training  than  was  delivered. 

1 

The  syllabus  and  missions  were  presented  in  an  organized  manner. 

1 

As  a  result  of  this  training,  I  have  improved  my  mission  planning  skills. 

1 

At  my  unit,  I  routinely  get  to  fly  against  realistic  threats. 

1 

I  have  improved  my  tactical  skills  as  a  result  of  the  DMO  experience. 

1 
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Strongly  Agree 


Somewhat  Agree 

Somewhat  Disagree 

Strongly  Disagree 

DMO  has  considerable  combat  mission  training  value. 

1 

2 

3 

4 

I  am  confident  that  I  can  perfonn  the  tasks  taught  in  this  training  program  in  the  field. 

1 

2 

3 

4 

There  are  a  sufficient  number  of  4Vx  flying  opportunities  at  my  unit. 

1 

2 

3 

4 

DMO  improved  my  understanding  of  critical  combat  skills. 

1 

2 

3 

4 

The  scenarios  were  well  designed. 

1 

2 

3 

4 

DMO  missions  and  engagements  accurately  represent  how  things  happen  in  the  operational 
world. 

1 

2 

3 

4 

I  improved  my  briefing  and  debriefing  skills. 

1 

2 

3 

4 

The  visual  databases  were  realistic. 

1 

2 

3 

4 

The  fidelity  of  the  DMO  environment  was  sufficient  to  achieve  the  desired  mission/training 
objectives. 

1 

2 

3 

4 

DMO  has  limited  military  training  value. 

1 

2 

3 

4 

Using  DMO  in  conjunction  with  live  flying  is  worthwhile. 

1 

2 

3 

4 

DMO  has  improved  my  combat  mission  readiness. 

1 

2 

3 

4 

I  would  like  to  see  DMO  operational  applications  expanded. 

1 

2 

3 

4 

This  training  provided  excellent  practice  on  3-1  communication. 

1 

2 

3 

4 

DMO  provides  valuable  experience  in  combat  mission  tactics. 

1 

2 

3 

4 

I  expected  this  training  to  be  a  valuable  experience. 

1 

2 

3 

4 

As  a  result  of  this  training,  I  have  improved  my  BVR  tactics. 

1 

2 

3 

4 

DMO  should  be  a  part  of  all  future  spin-up  exercises. 

1 

2 

3 

4 

I  feel  I  will  be  better  prepared  for  to  brief  and  lead  a  mission  due  to  this  training. 

1 

2 

3 

4 

I  am  confident  I  will  perform  better  as  a  fighter  pilot  /  controller  as  a  result  of  DMO. 

1 

2 

3 

4 

I  routinely  get  to  practice  tactics  in  multiple  aircraft  enviromnents  at  my  operational  unit. 

1 

2 

3 

4 

As  a  result  of  this  training,  I  have  improved  my  VID  tactics. 

1 

2 

3 

4 

The  threats  in  DMO  behaved  as  they  do  in  the  operational  environment 

1 

2 

3 

4 

DMO  has  helped  me  improve  my  team  coordination  skills. 

1 

2 

3 

4 

The  missions  and  engagements  were  very  challenging. 

1 

2 

3 

4 

The  DMO  missions  I  experienced  were  unrealistic. 

1 

2 

3 

4 

This  training  helped  me  improve  my  combat  situational  awareness. 

1 

2 

3 

4 
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On  this  last  page,  please  comment,  as  appropriate,  to  each  of  the  following  questions. 

1.  List  the  “top  five”  things  you  feel  were  beneficial  about  the  training  you  received  here  at  DMT.  Next 
to  each  item,  please  state  why  it  was  beneficial 

a.  _  Why?  _ 

b.  _  Why?  _ 

c.  _  Why?  _ 

d.  _  Why?  _ 

e.  _  Why?  _ 

2.  The  WIC  spin-up  represents  our  first  attempt  to  provide  a  sequenced,  learning  objectives-based 
syllabus  for  DMT.  Do  you  feel  this  is  an  appropriate  way  to  structure  complex  training  such  as  this? 
(If  applicable) 

Circle  one:  Definitely  Yes  Probably  Definitely  Not 

3.  What  recommendations  do  you  have  that  would  improve  the  Intel  portion  of  your  in-brief? 


4.  What  recommendations  do  you  have  that  could  improve  the  training  process  next  time? 


5.  Were  there  any  aspects  of  your  AFRL/HEA  DMT  experience  that  you  felt  led  to  unrealistic  or 
inappropriate  content  being  provided?  If  so,  please  provide  details  below. 


6.  If  you  could  improve  one  technical  capability  of  the  simulation  training  environment  at  AFRL/HEA 
what  would  it  be?  And  why? 
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This  page  intentionally  left  blank 
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APPENDIX  B: 
MEC  Experience  Survey 


Air-to-Air  MECsm  Ranking  and  Rating  Questionnaire 

MEC  Definitions 

1.  Plan/Prepare  for  Mission:  Reviews,  understands,  and  identifies  limitations  of  intelligence  and  all  mission- 
related  documents  (e.g.,  ATO,  ACO,  SPINS);  reviews  available  assets  for  communications/data  management 
during  mission  and  synergistic  combat  effects  of  other  assets;  conducts  route  and  target  area  risk,  and  own 
ship/flight  asset  analyses;  formulates  game  plan  to  include  maps,  comms,  ingress  and  egress,  timing,  mission 
materials,  DTC/DTM/MC  load;  conducts  contingency  planning,  including  strengths/weaknesses  of  game  plan; 
identifies  possible  problems  such  as  fallout  or  timing  problems  and  appropriate  reactions;  effectively  briefs 
game  plan  with  all  available  assets  including  any  joint  or  coalition  assets;  assembles  all  necessary  materials  to 
bring  on  mission. 

Start:  At  mission  tasking  (ATO  publication) 

End:  End  of  mission  brief 

Purpose:  Equip  and  organize  the  package 

2.  Force  Organization  MEC:  Organizes  forces  to  enable  combat  employment 

Starting  Point:  Force  organization  begins  when  the  force  enters  the  area  of  interest  and  initial  check-in 
with  Command  and  Control. 

Purpose:  Establish  communication  links  with  all  assets;  establish  force  structure  (e.g,.  position,  timing,  and 
relationship  to  package);  determine  any  known  force  limitations. 

End  Point:  At  the  establishment  of  the  CAP  or  at  the  push. 

3.  Detection  Phase  MEC:  Detects  factor  groups  in  the  area  of  responsibility. 

Starting  Point:  Detection  starts  when  the  flight  enters  the  vulnerability  period  or  push  time  and  is  in  a 
position  to  target  a  designated  airborne  entity  and  start  an  intercept.  From  an  ISR  perspective  detection  is  a 
continuous  process.  Follow-on  waves  or  groups  of  potential  targets  must  be  detected  in  area  of 
responsibility  (AOR)  to  maintain  SA  and  provide  triggers  for  subsequent  decisions. 

Purpose:  Process  information  necessary  to  trigger  the  commit  decision  on  entities  that  meet  commit 
criteria. 

End  Point:  Normally  when  the  flight  lead  makes  the  commit  decision.  Detection  is  actually  a  continuous 
process  for  the  entire  vulnerability  period. 

4.  Intercept  and  Targeting  Phase  MEC:  Intercepts  and  targets  factor  groups. 

Starting  Point:  Intercept  and  targeting  starts  at  the  commit.  Geometry  and  contact  ranges  may  cause  the 
two  parts  of  this  phase  to  blend  together.  Very  short  commits  may  not  have  an  intercept  portion  and  flights 
may  have  to  go  directly  to  targeting.  As  with  detection,  targeting  may  be  a  repetitive  action  if  multiple 
entities  and  waves  are  involved. 

Purpose:  Arrive  with  a  positional  advantage  and  in  a  position  to  employ  ordnance.  Deconflict  shots. 

End  Point:  When  engagement  criteria  are  met  in  terms  of  weapons  parameters,  ROE,  and  target 
identification. 

5.  Engagement  Phase  MEC:  Employs  ordnance  against  valid  hostile  targets  and/or  denies  enemy  weapons  IAW 
mission  objectives. 

Starting  Point:  Engagement  starts  at  weapons  employment/defensive  reaction.  Encompasses  ROE, 
weapons  employment  and  all  aspects  of  offensive  and  defensive  maneuvering. 
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Purpose:  Defeat/deny  enemy  forces  (offensively  and  defensively) 

End  Point:  Engagement  ends  when  the  fighter/ fl ight  has  achieved  the  engagement  objectives,  has  to 
disengage  due  to  fuel/ordnance  considerations,  or  has  successfully  negated  an  enemy  attack. 

6.  Assessment/Reconstitution  Phase  MEC:  Determines  and  initiates  appropriate  follow  on  actions  (e.g.,  the 
need/ability  to  engage  follow-on  forces,  return  to  the  CAP,  go  to  the  tanker,  RTB.) 

Starting  Point:  Assessment  starts  when  the  fighter/flight  has  negated  the  initial  threat  through  denial  or 
defeat.  Assumes  a  defensive  situation  has  been  resolved. 

Purpose:  Identifies  the  need  to  re-engage  the  initial  threat,  engage  subsequent  threats,  and  the  ability  to 
continue  to  defend  the  airspace.  Information  from  this  phase  used  to  decide  whether  to  initiate  follow-on 
intercept/targeting,  return  to  detection,  air  refuel,  or  return  to  base. 

End  Point:  This  phase  ends  with  the  initiation  of  a  follow-on  action. 

7.  Force  Orientation  MEC:  Remains  oriented  to  force  requirements. 

Starting  Point:  Force  orientation  begins  when  organization  has  been  accomplished,  initiation  of  the 
gameplan  and  reacting  with  and  in  response  to  each  other  in  an  inter-team  arrangement. 

Purpose:  Individual,  flight  and  force  management  and  orientation  during  execution  of  prescribed  mission. 
End  Point:  Package  elements  have  egressed,  conducted  post-strike  activities  such  as  refueling,  and  no 
longer  function  in  an  inter-team  arrangement. 

8.  Phase  Transition  MEC:  Recognizes  the  trigger  events/situations  that  require  a  shift  from  one  phase  to  the  next. 

Starting  Point:  Recognizing  trigger  events/situations  that  require  a  shift  from  one  phase  to  the  next  begins 
when  the  flight  enters  the  vulnerability  period  or  push  time  and  is  in  a  position  to  start  an  intercept  or  target 
a  designated  airborne  entity. 

Purpose:  Gathered  information  is  used  to  make  decisions  (e.g.,  commit,  engage,  and  follow-on  decisions) 
End  Point:  At  the  completion  of  the  mission. 


This  information  is  For  Official  Use  Only.  These  materials  support  the  Mission  Essential  Competency  SM  (MEC) 
development  effort.  Distribution  of  these  materials  to  individuals,  contractors,  or  other  agencies  that  are  not  directly 
involved  with  the  MEC  definition  process  or  work,  without  the  expressed  approval  from  Dr.  Winston  Bennett,  Warfighter 
Training  Research  Division  (AFRL/HEAS)  (480.988.6561  x297)  or  Mr.  Charles  Colegrove,  HQ  ACC/DOTO  (757-764- 
7785),  is  not  authorized  under  any  circumstances. 
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Air-to-Air  MEC  Ranking  and  Rating  Questionnaire 


Instructions:  Complete  this  survey  by  ranking  the  left-hand  column  and  rating  the  right- 
hand  column  according  to  the  directions  for  each  column. 


In  this  column,  using  numbers  1  to  8, 
rank  each  of  the  eight  MECs  in  terms  of 
your  preparedness  for  performing  each 
MEC  in  sustained  combat  operations. 
Use  “1”  for  the  MEC  for  which  you  have 
the  highest  level  of  preparedness,  “2”  for 
the  MEC  for  which  you  have  the  next 
highest  level  of  preparedness,  and  so 
forth,  through  number  “8”.  Use  each 
number,  1  through  8  only  once. 


Note:  you  may  feel  prepared  to  perform 
each  MEC,  but  please  rank  them  1  to  8 


In  this  column,  using  the  scale 
below,  rate  each  MEC: 

1=1  am  not  ready  to  perform  this  area  in  a  non- 
permissive  environment. 

2  =  I’m  ready  to  go,  however  I’d  like  to  get  a 

substantial  amount  of  additional  experience 
in  this  area. 

3  =  I’m  ready  to  go,  however  I’d  like  to  get  a  fair 

amount  of  additional  experience  in  this  area. 

4  =  I’m  ready  to  go,  however  I’d  like  to  get  a 

little  additional  experience  in  this  area. 

5  =  I’m  ready  to  go,  and  I  need  no  additional 

experience  in  this  area. 


RANK 

Mission  Essential  Competency 
(MEC) 

RATING 

Plan/Prepare  for  Mission 

Force  Organization 

Detection  Phase 

Intercept  and  Targeting  Phase 

Engagement  Phase 

Assessment/Reconstitution  Phase 

Force  Orientation 

Phase  Transition 
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Research  Id  #: 
Date: 


Air-to-Air  MECsm  Experiences  and  Environments  (Ratings) 

Questionnaire 


Definitions  of  Air-to-Air  Current  or  Recent  Learning  Environments 

1.  RAP  Flying  Events:  All  Ready  Aircrew  Program  flying  events,  with  the  exception  of 
Flag/CFTR 

2.  RAP  Flying  Events:  Flag/CFTRs 

•  Flag  -  Red,  Green,  or  Maple  Flag  participation 

•  CFTR  -  Scenarios  employing  multiple  flights  of  the  same  or  different  types  of 
aircraft,  each  under  the  direction  of  its  own  flight  leader,  performing  the  same  or 
different  roles 

3.  MTC/FMT  -  All  simulator  missions  in  the  Mission  Training  Center  and  Full  Mission 
Trainer 

4.  ONW/OSW  -  All  experiences  in  Operations  Northern  or  Southern  Watch 

5.  UTD  -  Training  conducted  in  this  high-fidelity  squadron  level  Unit  Training  Device 

6.  AFRL/Mesa  -  Training  conducted  at  the  Air  Force  Research  Lab  in  Mesa.  This  is  a  highly 
integrated  set  of  four  F 16  simulators 

7.  WTT/DTT  -  Training  conducted  in  the  Weapons  and  Tactics  Trainer  and  Desk  Top  Trainer 

8.  Sustained  Combat  Operations  -  Missions  flown  during  declared  hostilities  such  as  OAF 
and  Desert  Storm.  Does  not  include  ONW  or  OSW 


This  information  is  For  Official  Use  Only.  These  materials  support  the  Mission  Essential  Competency  (MEC)sm 
development  effort.  Distribution  of  these  materials  to  individuals,  contractors,  or  other  agencies  that  are  not  directly 
involved  with  the  MEC  definition  process  or  work,  without  the  expressed  approval  from  Dr.  Winston  Bennett, 
Warfighter  Readiness  Research  Division  (AFRL/HEAS)  (480.988.6561  x297)  or  Mr.  Charles  Colegrove,  HQ 
ACC/DOTO  (757-764-7785),  is  not  authorized  under  any  circumstances. 
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Air-to-Air  MECsm  Experience  and  Environments  (Ratings) 

Instructions: 

The  matrix  below  lists  learning  environments  across  the  top.  Please  refer  to  the  previous  page  for  a  more  complete  description  of  each 
environment. 

Common  experiences  (operations  under  various  conditions)  are  listed  down  the  left-hand  side. 

We  are  trying  to  identify  whether  experiences  can  realistically  be  trained  in  various  environments.  Please  rate  to  what  extent  it  is 
reasonably  possible  to  provide  each  experience  in  each  environment. 

0  =  Not  at  all/Does  Not  Apply 

1  =  To  a  Slight  Extent 

2  =  To  a  Moderate  Extent 

3  =  To  a  Substantial  Extent 

4  =  To  a  Great  Extent 


To  what  extent  can  operations... 

Environments 

. under 

these 

conditions 

be  experienced 
in  these 
environments? 

c> 

RAP  Flying  Events 

MTC/ 

FMT 

ONW/ 

osw 

UTD 

AFRL/ 

Mesa 

WTT/DTT 

Sustained  Combat 
Ops  (e.g.,  OAF, 
Desert  Storm) 

All  except 
Flag/CFTR 

Flag/CFTR 

13  l 

1.  Restricted  weapons  load  (e.g.,  due  to 
previous  weapons  employment, 
incompleted  reload,  or  WRM 
limitations) 

2.  Limited  fuel  remaining  (e.g.,  due  to 
increased  fuel  consumption,  low  fuel 
remaining,  lack  of  tanker  support,  or 
the  inability  to  inflight  refuel) 

3.  Operating  area  restrictions  (e.g., 
geographic,  altitude,  or  political) 

4.  Restrictions  to  visibility  (e.g.,  haze) 

5.  Visual  illusions 

6.  Marginal/minimal  cloud  clearance 
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We  are  trying  to  identify  whether  experiences  can  realistically  be  trained  in  various  environments.  Please  rate  to  what  extent  it  is 
reasonably  possible  to  provide  each  experience  in  each  environment. 

0  =  Not  at  all/Does  Not  Apply 

1  =  To  a  Slight  Extent 

2  =  To  a  Moderate  Extent 

3  =  To  a  Substantial  Extent 

4  =  To  a  Great  Extent 


To  what  extent  can  operations... 

Environments 

. under 

be  experienced 

> 

RAP  Flying  Events 

Sustained  Combat 
Ops  (e.g.,  OAF, 
Desert  Storm) 

these 

conditions 

environments? 

All  except 
Flag/CFTR 

Flag/CFTR 

MTC/ 

FMT 

ONW/ 

osw 

UTD 

AFRL/ 

Mesa 

WTT/DTT 

7. 

Daytime  employment 

8. 

Dusk  employment 

9. 

Night  employment 

10.  Mountainous  terrain 

11 

.  G-induced  physical  limitations 

12.  Degraded  comm.,  (e.g.,  due  to 

ownship  systems  malfunction,  another 
aircraft’s  malfunction,  or  the  inability 
to  use  HQ  or  secure  voice  by  one  or 
more  aircraft) 

13.  Degraded  nav. 

14.  Degraded  weapons  employment 
capability  (e.g.,  fire  control,  RWR, 
missiles) 

15.  Battle  damage  (e.g.,  operations  with 
battle  damage  of  ownship  or  of  another 
aircraft  in  the  formation) 

16.  Supersonic  employment 

34 


We  are  trying  to  identify  whether  experiences  can  realistically  be  trained  in  various  environments.  Please  rate  to  what  extent  it  is 
reasonably  possible  to  provide  each  experience  in  each  environment. 

0  =  Not  at  all/Does  Not  Apply 

1  =  To  a  Slight  Extent 

2  =  To  a  Moderate  Extent 

3  =  To  a  Substantial  Extent 

4  =  To  a  Great  Extent 


To  what  extent  can  operations... 

Environments 

. under 

be  experienced 

> 

RAP  Flying  Events 

Sustained  Combat 
Ops  (e.g.,  OAF, 
Desert  Storm) 

these 

conditions 

environments? 

All  except 
Flag/CFTR 

Flag/CFTR 

MTC/ 

FMT 

ONW/ 

osw 

UTD 

AFRL/ 

Mesa 

WTT/DTT 

17.  A  fulf range  of  adversary  air  threat  and 
mix  (e.g.,  various  limitations  in 
maneuvering,  tactics,  and  weapons) 

18.  A  full  range  of  adversary  ground  type 
and  mix  (including  old  and  latest 
threats) 

19.  Operations  with  friendly  IADS 

20.  Ownship  and  friendly  electronic 
counter  measures 

21 

.  Operations  against  a  threat  using 
chaff/ flare 

22.  Using  chaff/flare  to  deny/defeat  enemy 
radar/weapons. 

23.  Operations  against  adversary 
jam/spoofing 

comm. 

24.  Operations  against  air  or  ground 
adversary  j  amming 
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We  are  trying  to  identify  whether  experiences  can  realistically  be  trained  in  various  environments.  Please  rate  to  what  extent  it  is 
reasonably  possible  to  provide  each  experience  in  each  environment. 

0  =  Not  at  all/Does  Not  Apply 

1  =  To  a  Slight  Extent 

2  =  To  a  Moderate  Extent 

3  =  To  a  Substantial  Extent 

4  =  To  a  Great  Extent 


To  what  extent  can  operations... 

Environments 

. under 

these 

conditions 

be  experienced 
in  these 
environments? 

> 

RAP  Flying  Events 

MTC/ 

FMT 

ONW/ 

osw 

UTD 

AFRL/ 

Mesa 

WTT/DTT 

Sustained  Combat 
Ops  (e.g.,  OAF, 
Desert  Storm) 

All  except 
Flag/CFTR 

Flag/CFTR 

25.  ROrTHmitations  (e.g.,  operations  in  an 
environment  that  has  restrictive  ID 
requirements  (other  than  BVR 
weapons  free)  such  as  VID,  PID, 
hostile  declaration  required  by 
offboard  sources) 

26.  Fatigue/time  on  task  (e.g.,  long  range 
force  employment) 

27.  Task  saturation 

28.  Limited  time  to  act/react  to  situation 

29.  Radar  search  responsibilities 

30.  Targeting  and  sorting  responsibilities 

31.  Air  refueling 

32.  Live  weapons  employment  (e.g., 
WSEP,  combat) 

33.  Simulated  weapons  employment  (e.g., 
training) 
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We  are  trying  to  identify  whether  experiences  can  realistically  be  trained  in  various  environments.  Please  rate  to  what  extent  it  is 
reasonably  possible  to  provide  each  experience  in  each  environment. 

0  =  Not  at  all/Does  Not  Apply 

1  =  To  a  Slight  Extent 

2  =  To  a  Moderate  Extent 

3  =  To  a  Substantial  Extent 

4  =  To  a  Great  Extent 


To  what  extent  can  operations... 

Environments 

. under 

these 

conditions 

be  experienced 
in  these 
environments? 

> 

RAP  Flying  Events 

MTC/ 

FMT 

ONW/ 

osw 

UTD 

AFRL/ 

Mesa 

WTT/DTT 

Sustained  Combat 
Ops  (e.g.,  OAF, 
Desert  Storm) 

All  except 
Flag/CFTR 

Flag/CFTR 

34.  Various  initial  conditions  (e.g.,  perch 
setups,  CAP  &  tap,  and  flights  on 
ranges) 

35.  Emergency  procedures 

36.  Formation  responsibilities  (e.g., 
position,  visual  lookout) 

37.  Lost  mutual  support 

38.  Dynamic  retasking/scramble 
operations 

39.  Various  employment  altitudes  (e.g., 
low,  med,  high) 

40.  1:1  Force  ratio  (e.g.,  2  v  2,  4  v  4) 

41.  1:2  Force  ratio  (e.g.,  4  v  8) 

42.  1:3+  Force  ratio  (e.g.,  4  v  12+) 

43.  OCA  escort  missions 

44.  OCA  sweep  missions 
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We  are  trying  to  identify  whether  experiences  can  realistically  be  trained  in  various  environments.  Please  rate  to  what  extent  it  is 
reasonably  possible  to  provide  each  experience  in  each  environment. 

0  =  Not  at  all/Does  Not  Apply 

1  =  To  a  Slight  Extent 

2  =  To  a  Moderate  Extent 

3  =  To  a  Substantial  Extent 

4  =  To  a  Great  Extent 


To  what  extent  can  operations... 

Environments 

. under 

be  experienced 

> 

RAP  Flying  Events 

Sustained  Combat 
Ops  (e.g.,  OAF, 
Desert  Storm) 

these 

conditions 

environments? 

All  except 
Flag/CFTR 

Flag/CFTR 

MTC/ 

FMT 

ONW/ 

osw 

UTD 

AFRL/ 

Mesa 

WTT/DTT 

45.  Employment  with  various  package 
requirements  (e.g.,  operations  in 
conjunction  with  different  aircraft  and 
supporting  assets,  includes 
determining  and  operating  with 
minimum  package  requirements) 
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APPENDIX  C: 

Time-Based  Analysis  of  DMO  Reaction  Results 


Table  14  Number  of  Participants  per  Group  Over  Time 


Participant  Groups 

Over  Time 

Number  of 

Participants 

January  2002-June  2002 

43 

July  2002-December  2002 

52 

January  2003-June  2003 

56 

July  2003-December  2003 

64 

January  2004-June  2004 

64 

July  2002-October  2004 

47 

Table  2  Overall  Effect  of  Time  on  DMO  Expectations 


Source 

Type  III  Sum 
of  Squares 

Degrees  of 
Freedom 

Mean 

Square 

F 

Sig. 

Corrected  Model 

2.469a 

5 

.494 

3.676 

.003 

Intercept 

4201.996 

1 

4201.996 

31277.871 

.000 

Participant  Groups  Over  Time 

2.469 

5 

.494 

3.676 

.003 

Error 

42.990 

320 

.134 

Total 

4327.249 

326 

Corrected  Total 

45.459 

325 

a:  R  Squared  =  .054  (Adjusted  R  Squared  =  .040) 
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Table  3  DMO  Expectations  Contrast  Results 


Participant  Group  Polynomial  Contrast 

HUH 

Linear 

Contrast  Estimate 

Hypothesized  Value 

0 

Difference  (Estimate-  Hypothesized) 

.000 

Standard  Error 

.053 

Significance 

.996 

95%  Confidence  Interval  Lower  Bound 

-.104 

for  Difference  Upper  Bound 

.104 

Quadratic 

Contrast  Estimate 

.017 

Hypothesized  Value 

0 

Difference  (Estimate-  Hypothesized) 

.017 

Standard  Error 

.052 

Significance 

.739 

95%  Confidence  Interval  Lower  Bound 

-.085 

for  Difference  Upper  Bound 

.119 

Cubic 

Contrast  Estimate 

.155 

Hypothesized  Value 

0 

Difference  (Estimate-  Hypothesized) 

.155 

Standard  Error 

.050 

Significance 

.002 

95%  Confidence  Interval  Lower  Bound 

.056 

for  Difference  Upper  Bound 

.253 
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Figure  1  Overall  Effect  of  Time  on  DMO  Expectations 
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Table  4_Overall  Effect  of  Time  on  DMO  Opinions 


Source 

Type  III  Sum 
of  Squares 

Degrees  of 
Freedom 

Mean 

Square 

F 

Sig. 

Corrected  Model 

.490a 

5 

.098 

1.361 

.239 

Intercept 

4419.41 

1 

4419.413 

61321.615 

.000 

Participant  Groups  Over  Time 

.490 

5 

.098 

1.361 

.239 

Error 

23.062 

320 

.072 

Total 

4540.541 

326 

Corrected  Total 

23.552 

325 

a:  R  Squared  =  .021  (Adjusted  R  Squared  =  .006) 


Table  5  DMO  Opinions  Contrast  Results 


Participant  Group  Polynomial  Contrast 

DMO 

Expectations 

Linear 

Contrast  Estimate 

.062 

Hypothesized  Value 

0 

Difference  (Estimate-  Hypothesized) 

.062 

Standard  Error 

.039 

Significance 

.112 

95%  Confidence  Interval  Lower  Bound 

-.015 

for  Difference  Upper  Bound 

.138 

Quadratic 

Contrast  Estimate 

-.009 

Hypothesized  Value 

0 

Difference  (Estimate-  Hypothesized) 

-.009 

Standard  Error 

.038 

Significance 

.814 

95%  Confidence  Interval  Lower  Bound 

-.084 

for  Difference  Upper  Bound 

.066 

Cubic 

Contrast  Estimate 

.069 

Hypothesized  Value 

0 

Difference  (Estimate-  Hypothesized) 

.069 

Standard  Error 

.037 

Significance 

.060 

95%  Confidence  Interval  Lower  Bound 

-.003 

for  Difference  Upper  Bound 

.141 
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Mean  DMO  Opinion  Rating 


DMO  Opinions 


Figure  2  Overall  Effect  of  Time  on  DMO  Opinions 


Table  6  Overall  Effect  of  Time  on  DMO  Scenario  Characteristics 


Source 

Type  III  Sum 
of  Squares 

Degrees  of 
Freedom 

Mean 

Square 

F 

Sig. 

Corrected  Model 

1.847a 

5 

.369 

1.694 

.136 

Intercept 

3185.901 

1 

3185.901 

14613.972 

.000 

Participant  Groups  Over  Time 

1.847 

5 

.369 

1.694 

.136 

Error 

69.761 

320 

.218 

Total 

3334.333 

326 

Corrected  Total 

71.608 

325 

a:  R  Squared  =  .026  (Adjusted  R  Squared  =  .011) 
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Table  7  DMO  Scenario  Characteristics  Contrast  Results 


Participant  Group  Polynomial  Contrast 

DMO 

Expectations 

Linear 

Contrast  Estimate 

.139 

Hypothesized  Value 

0 

Difference  (Estimate-  Hypothesized) 

.139 

Standard  Error 

.067 

Significance 

.041 

95%  Confidence  Interval  Lower  Bound 

.006 

for  Difference  Upper  Bound 

.271 

Quadratic 

Contrast  Estimate 

-.055 

Hypothesized  Value 

0 

Difference  (Estimate-  Hypothesized) 

-.055 

Standard  Error 

.066 

Significance 

.405 

95%  Confidence  Interval  Lower  Bound 

-.185 

for  Difference  Upper  Bound 

.075 

Cubic 

Contrast  Estimate 

.086 

Hypothesized  Value 

0 

Difference  (Estimate-  Hypothesized) 

.086 

Standard  Error 

.064 

Significance 

.177 

95%  Confidence  Interval  Lower  Bound 

-.039 

for  Difference  Upper  Bound 

.212 
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Mean  DMO  Scenario  Characteristics  Rating 


DMO  Scenario  Characteristics 


4 


3.5 


3 


2.5 


2 


1.5 


1 


Figure  3  Overall  Effect  of  Time  on  DMO  Scenario  Characteristics 


Table  8  Overall  Effect  of  Time  on  General  Comments  About  the  DMO  Environment 


a:  R  Squared  =  .023  (Adjusted  R  Squared  =  .008) 


Jan. '02  -  June'02  July '02  -  Dec. '02  Jan.  '03  -  June  '03  July '03  -  Dec. '03  Jan '04  -  June '04  July '04  -  Oct. '04 

Participant  Groups  Over  Time 
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Table  9  General  Comments  about  the  DMO  Environment  Contrast  Results 


Participant  Group 

Polynomial  Contrast 

Linear 

Contrast  Estimate 

.062 

Hypothesized  Value 

0 

Difference  (Estimate-  Hypothesized) 

.062 

Standard  Error 

.096 

Significance 

.523 

95%  Confidence  Interval  Lower  Bound 

-.128 

for  Difference  Upper  Bound 

.251 

Quadratic 

Contrast  Estimate 

-.196 

Hypothesized  Value 

0 

Difference  (Estimate-  Hypothesized) 

-.196 

Standard  Error 

.094 

Significance 

.039 

95%  Confidence  Interval  Lower  Bound 

-.381 

for  Difference  Upper  Bound 

-.01 

Cubic 

Contrast  Estimate 

.020 

Hypothesized  Value 

0 

Difference  (Estimate-  Hypothesized) 

.020 

Standard  Error 

.091 

Significance 

.829 

95%  Confidence  Interval  Lower  Bound 

-.159 

for  Difference  Upper  Bound 

.199 
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Mean  DMO  General  Comments  Rating 


DMO  General  Comments 


Figure  4  Overall  Effect  of  Time  on  General  Comments  about  the  DMO  Environment 

Table  10  Overall  Effect  of  Time  on  Home  Unit  Conditions 


Source 

Type  III  Sum 
of  Squares 

Degrees  of 
Freedom 

Mean 

Square 

F 

Sig. 

Corrected  Model 

2.479a 

5 

.496 

1.746 

.124 

Intercept 

3754.029 

1 

3754.029 

13215.916 

.000 

Participant  Groups  Over  Time 

2.479 

5 

.496 

1.746 

.124 

Error 

90.897 

320 

.284 

Total 

3936.879 

326 

Corrected  Total 

93.377 

325 

a:  R  Squared  =  .023  (Adjusted  R  Squared  =  .008) 
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Table  11  Home  Unit  Conditions  Contrast  Results 


Participant  Group  Polynomial  Contrast 

DMO 

Expectations 

Linear 

Contrast  Estimate 

.123 

Hypothesized  Value 

0 

Difference  (Estimate-  Hypothesized) 

.123 

Standard  Error 

.077 

Significance 

.110 

95%  Confidence  Interval  Lower  Bound 

-.028 

for  Difference  Upper  Bound 

.275 

Quadratic 

Contrast  Estimate 

-.081 

Hypothesized  Value 

0 

Difference  (Estimate-  Hypothesized) 

-.081 

Standard  Error 

.075 

Significance 

.283 

95%  Confidence  Interval  Lower  Bound 

-.230 

for  Difference  Upper  Bound 

.067 

Cubic 

Contrast  Estimate 

.142 

Hypothesized  Value 

0 

Difference  (Estimate-  Hypothesized) 

.142 

Standard  Error 

.073 

Significance 

.051 

95%  Confidence  Interval  Lower  Bound 

-.001 

for  Difference  Upper  Bound 

.286 
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Mean  DMO  Home  Unit  Conditions  Rating 


Home  Unit  Conditions 


Figure  5  Overall  Effect  of  Time  on  Home  Unit  Conditions 


Table  12  Overall  Effect  of  Time  on  DMO  Overall  Training  Value 


Source 

Type  III  Sum 
of  Squares 

Degrees  of 
Freedom 

Mean 

Square 

F 

Si£. 

Corrected  Model 

.533a 

5 

.107 

1.560 

.171 

Intercept 

4348.941 

1 

4348.941 

63706.126 

.000 

Participant  Groups  Over  Time 

.533 

5 

.107 

1.560 

.171 

Error 

21.845 

320 

.068 

Total 

4462.975 

326 

Corrected  Total 

22.378 

325 

a:  R  Squared  =  .024  (Adjusted  R  Squared  =  .009) 
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Table  13  DMO  Overall  Training  Value  Contrast  Results 


Participant  Group  Polynomial  Contrast 

DMO 

Expectations 

Linear 

Contrast  Estimate 

-.004 

Hypothesized  Value 

0 

Difference  (Estimate-  Hypothesized) 

-.004 

Standard  Error 

.038 

Significance 

.920 

95%  Confidence  Interval  Lower  Bound 

-.078 

for  Difference  Upper  Bound 

.070 

Quadratic 

Contrast  Estimate 

-.004 

Hypothesized  Value 

0 

Difference  (Estimate-  Hypothesized) 

-.004 

Standard  Error 

.037 

Significance 

.920 

95%  Confidence  Interval  Lower  Bound 

-.076 

for  Difference  Upper  Bound 

.069 

Cubic 

Contrast  Estimate 

.093 

Hypothesized  Value 

0 

Difference  (Estimate-  Hypothesized) 

.093 

Standard  Error 

.036 

Significance 

.009 

95%  Confidence  Interval  Lower  Bound 

.023 

for  Difference  Upper  Bound 

.163 
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Mean  DMO  Training  Value  Rating 


DMO  Overall  Training  Value 


Figure  6.  Overall  Effect  of  Time  on  DMO  Overall  Training  Value 


Table  14  Overall  Effect  of  Time  on  Syllabus/Mission  Flow 


Source 

Type  III  Sum 
of  Squares 

Degrees  of 
Freedom 

Mean 

Square 

F 

Sig. 

Corrected  Model 

A9T 

5 

.099 

1.370 

.235 

Intercept 

3666.391 

1 

3666.391 

50499.969 

.000 

Participant  Groups  Over  Time 

.497 

5 

.099 

1.370 

.235 

Error 

23.233 

320 

.073 

Total 

3774.291 

326 

Corrected  Total 

23.730 

325 

a:  R  Squared  =  .021  (Adjusted  R  Squared  =  .006) 
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Table  15  Syllabus/Mission  Flow  Contrast  Results 


Participant  Group  Polynomial  Contrast 

DMO 

Expectations 

Linear 

Contrast  Estimate 

.001 

Hypothesized  Value 

0 

Difference  (Estimate-  Hypothesized) 

.001 

Standard  Error 

.039 

Significance 

.980 

95%  Confidence  Interval  Lower  Bound 

-.076 

for  Difference  Upper  Bound 

.078 

Quadratic 

Contrast  Estimate 

-.057 

Hypothesized  Value 

0 

Difference  (Estimate-  Hypothesized) 

-.057 

Standard  Error 

.038 

Significance 

.138 

95%  Confidence  Interval  Lower  Bound 

-.132 

for  Difference  Upper  Bound 

.018 

Cubic 

Contrast  Estimate 

.073 

Hypothesized  Value 

0 

Difference  (Estimate-  Hypothesized) 

.073 

Standard  Error 

.037 

Significance 

.049 

95%  Confidence  Interval  Lower  Bound 

.000 

for  Difference  Upper  Bound 

.145 
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Mean  Syllabus/Mission  Flow  Rating 


DMO  Syllabus/Mission  Flow 


Participant  Groups  Over  Time 


Figure  7.  Overall  Effect  of  Time  on  Syllabus/Mission  Flow 
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APPENDIX  D: 

MEC  Experience  by  Environment  Results 


To  what  extent  can  operations... 

Environments 

...under  these 

be  experienced 

0 

RAP  Flying 
Events 

Sustained 

Combat 

All 

Flag/ 

CFTR 

MTC/ 

FMT 

ONW/ 

osw 

UTD 

AFRL/ 

Mesa 

WTT/ 

DTT 

Ops  (e.g., 
OAF, 
Desert 
Storm) 

except 

Flag/ 

CFTR 

1.  Restricted  weapons  load  (e.g., 
due  to  previous  weapons 
employment,  incompleted 
reload,  or  WRM  limitations) 

2.56 

2.47 

2.34 

.94 

2.63 

3.63 

1.31 

1.87 

2.  Limited  fuel  remaining  (e.g., 
due  to  increased  fuel 
consumption,  low  fuel 
remaining,  lack  of  tanker 
support,  or  the  inability  to 
inflight  refuel) 

2.09 

2.31 

2.19 

1.41 

2.44 

3.34 

1.16 

2.12 

3.  Operating  area  restrictions  (e.g., 
geographic,  altitude,  or  political) 

2.53 

2.69 

1.75 

1.75 

1.72 

2.75 

.94 

2.47 

4.  Restrictions  to  visibility  (e.g., 
haze) 

2.19 

2.00 

1.78 

1.31 

2.25 

2.63 

1.16 

1.94 

5.  Visual  illusions 

2.09 

1.84 

1.56 

1.12 

1.81 

2.31 

.97 

1.88 

6.  Marginal/minimal  cloud 
clearance 

2.38 

1.91 

1.75 

1.19 

2.19 

2.53 

1.03 

1.75 

7.  Daytime  employment 

3.00 

2.81 

2.00 

1.72 

2.50 

3.22 

1.50 

3.28 

8.  Dusk  employment 

2.19 

2.19 

1.63 

1.22 

2.09 

2.63 

1.06 

2.12 

9.  Night  employment 

2.84 

2.69 

1.66 

1.47 

2.03 

2.19 

1.09 

2.22 

10.  Mountainous  terrain 

2.19 

2.72 

1.56 

.78 

1.87 

2.47 

.75 

1.59 

1 1 .  G-induced  physical  limitations 

2.91 

2.50 

.25 

.69 

.75 

.84 

.25 

1.53 

12.  Degraded  comm,  (e.g.,  due  to 
ownship  systems  malfunction, 
another  aircraft's  malfunction,  or 
the  inability  to  use  HQ  or  secure 
voice  by  one  or  more  aircraft) 

2.38 

2.69 

1.06 

1.28 

1.59 

1.84 

.78 

2.00 

13.  Degraded  nav. 

1.84 

1.81 

1.53 

.87 

2.31 

2.00 

.84 

1.28 

14.  Degraded  weapons  employment 
capability  (e.g.,  fire  control, 
RWR,  missiles) 

1.78 

1.88 

1.75 

1.09 

2.50 

2.69 

1.09 

1.44 

15.  Battle  damage  (e.g.,  operations 
with  battle  damage  of  ownship 
or  of  another  aircraft  in  the 
formation) 

1.09 

1.03 

1.41 

.78 

1.81 

2.03 

.88 

1.63 
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To  what  extent  can  operations... 

Environments 

...under  these 

be  experienced 

RAP  Flying 
Events 

MTC/ 

FMT 

ONW/ 

osw 

UTD 

AFRL/ 

Mesa 

WTT/ 

DTT 

Sustained 
Combat 
Ops  (e.g., 
OAF, 
Desert 
Storm) 

All 

except 

Flag/ 

CFTR 

Flag/ 

CFTR 

16.  Supersonic  employment 

1.66 

2.28 

2.31 

1.31 

2.69 

3.06 

1.47 

1.84 

17.  A  full  range  of  adversary  air 
threat  and  mix  (e.g.,  various 
limitations  in  maneuvering, 
tactics,  and  weapons) 

1.47 

2.66 

1.84 

.78 

2.25 

3.41 

1.06 

1.44 

18.  A  full  range  of  adversary  ground 
type  and  mix  (including  old  and 
latest  threats) 

1.66 

2.72 

1.78 

1.13 

2.19 

2.72 

1.03 

1.72 

19.  Operations  with  friendly  IADS 

1.03 

2.22 

1.38 

1.31 

1.69 

3.16 

.84 

1.94 

20.  Ownship  and  friendly  electronic 
counter  measures 

2.06 

2.66 

1.56 

1.25 

1.97 

2.56 

.94 

1.91 

21.  Operations  against  a  threat  using 
chaff/ flare 

2.56 

2.88 

1.50 

1.78 

1.91 

2.78 

1.03 

1.75 

22.  Using  chaff/flare  to  deny/defeat 
enemy  radar/weapons. 

2.47 

2.75 

1.81 

1.31 

2.47 

3.06 

1.09 

1.94 

23.  Operations  against  adversary 
comm  jam/spoofing 

1.00 

2.19 

.94 

.72 

1.53 

1.59 

.50 

1.41 

24.  Operations  against  air  or  ground 
adversary  j  amming 

1.06 

2.37 

.97 

.66 

1.19 

1.78 

.62 

1.59 

25.  ROE  limitations  (e.g., 

operations  in  an  environment 
that  has  restrictive  ID 
requirements  (other  than  BVR 
weapons  free)  such  as  VID,  PID, 
hostile  declaration  required  by 
offboard  sources) 

2.34 

2.66 

1.62 

1.59 

1.81 

2.78 

.97 

2.28 

26.  Fatigue/time  on  task  (e.g.,  long 
range  force  employment) 

1.47 

1.84 

.69 

1.34 

1.00 

1.47 

.91 

2.37 

27.  Task  saturation 

2.34 

2.78 

1.69 

1.09 

1.78 

3.19 

.84 

2.12 

28.  Limited  time  to  act/react  to 
situation 

2.41 

2.66 

1.72 

1.16 

2.09 

3.16 

1.16 

2.09 

29.  Radar  search  responsibilities 

2.91 

3.03 

1.94 

1.34 

2.34 

3.44 

1.16 

1.94 

30.  Targeting  and  sorting 
responsibilities 

2.84 

3.00 

1.69 

1.34 

2.22 

3.19 

1.03 

1.87 

3 1 .  Air  refueling 

2.44 

2.47 

.41 

1.56 

.63 

.84 

.66 

2.47 

32.  Live  weapons  employment  (e.g., 
WSEP,  combat) 

1.47 

1.91 

.88 

1.19 

.81 

1.69 

.53 

2.28 
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To  what  extent  can  operations... 

Environments 

...under  these 

be  experienced 

RAP  Flying 
Events 

Sustained 

Combat 

All 

Flag/ 

CFTR 

MTC/ 

FMT 

ONW/ 

osw 

UTD 

AFRL/ 

Mesa 

WTT/ 

DTT 

Ops  (e.g., 
OAF, 
Desert 
Storm) 

o 

except 

Flag/ 

CFTR 

33.  Simulated  weapons  employment 
(e.g.,  training) 

2.84 

2.50 

2.06 

.72 

2.25 

3.03 

1.22 

1.06 

34.  Various  initial  conditions  (e.g., 
perch  setups,  CAP  &  tap,  and 
flights  on  ranges) 

2.53 

1.88 

1.62 

.53 

2.03 

2.62 

.84 

.88 

35.  Emergency  procedures 

1.47 

1.34 

1.66 

.75 

2.94 

2.00 

.75 

1.16 

36.  Formation  responsibilities  (e.g., 
position,  visual  lookout) 

3.12 

2.91 

1.34 

1.69 

1.00 

2.88 

.97 

2.34 

37.  Lost  mutual  support 

2.69 

2.66 

1.44 

.88 

1.16 

2.91 

.91 

1.75 

38.  Dynamic  retasking/scramble 
operat2.56ions 

1.78 

1.97 

1.31 

1.22 

1.56 

2.16 

1.06 

2.12 

39.  Various  employment  altitudes 
(e.g.,  low,  med,  high) 

2.69 

2.63 

1.75 

.94 

2.19 

3.16 

.94 

1.47 

40.  1:1  Force  ratio  (e.g.,  2  v  2,  4  v 

4) 

2.59 

2.31 

1.84 

.59 

1.72 

3.56 

.88 

.94 

41.  1:2  Force  ratio  (e.g.,  4  v  8) 

1.22 

2.63 

1.69 

.47 

1.47 

3.63 

.66 

.66 

42.  1:3+ Force  ratio  (e.g.,  4  v  12+) 

.66 

1.97 

1.66 

.44 

1.38 

3.66 

.59 

.63 

43.  OCA  escort  missions 

1.72 

2.16 

1.53 

.78 

1.41 

3.16 

.72 

1.03 

44.  OCA  sweep  missions 

1.75 

2.25 

1.50 

.63 

1.41 

3.22 

.69 

1.13 

45.  Employment  with  various 
package  requirements  (e.g., 
operations  in  conjunction  with 
different  aircraft  and  supporting 
assets,  includes  determining  and 
operating  with  minimum 
package  requirements) 

1.69 

2.91 

1.09 

1.13 

1.06 

2.47 

.81 

2.28 

OVERALL  MEANS,  STANDARD 
ERROR 

2.09,  .17 

2.37, 

.20 

1.54, 

.23 

1.08, 

.18 

1.85, 

.18 

2.65, 

.15 

.93,  .19 

1.74,  .23 

57 


This  page  intentionally  left  blank. 


58 


